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PLURIBUS DOCUMENT 2: SYSTEM HANDBOOK 
PREFACE 


"Pluribus Document 2: System Handbook" is one of a set of nine 
which, taken together, provide complete documentation of the 
Pluribus line of computer systems. In the present document, 
Part 1, entitled "Guide to Documentation," gives an overview of 
the entire set. Part 2, "System Description," contains an ex- 
tensive discussion of the Pluribus line and the ways in which it 
can be used. This system description is the primary text for 
anyone seeking familiarity with the Pluribus, although, of course, 
there are many details which can only be found elsewhere in the 
set. Part 3 is a glossary of specialized Pluribus terms used 
throughout the set. Part 4 is an index to the present document. 
Part 5 contains reprints of several papers relevant to the 


Pluribus... 
Of the five parts of "Pluribus Document 2," parts 1, 2, 3, and 


5 are presently included here; part 4 is in production and will 


be added when it becomes ready. 
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PART 1: GUIDE TO DOCUMENTATION 


GUIDE TO DOCUMENTATION 


Update History: 


Originally written - February 1975 
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The Pluribus line of computer systems is documented in 
a series of nine volumes entitled as follows: 


"Pluribus Document 1: Overview," BBN Report No. 2999 
"Pluribus Document 2: System Handbook," BBN Report 

No. 2930 | 
"Pluribus Document 3: Configurator," BBN Report No. 3000 
"Pluribus Document 4: Basic Software," BBN Report No. 3001 
"Pluribus Document 5: Advanced Software," BBN Report 

No. 2391 | 
"Pluribus Document 6: Functional Specifications," 

BBN Report No. 3002 | : 
"Pluribus Document 7: Construction," BBN Report No. 3003 
"Pluribus Document 8: Card Testing," BBN Report No. 3004 
"Pluribus Document 9: System Integration," BBN Report 

No. 3005 Ge Se "oe * | 


The set of documents taken as a whole is intended to cover 
all aspects of the Pluribus; e.g. , the decision to use a Pluribus, 
the design of systems involving the Pluribus, programming the 
Pluribus, actually fabricating the Pluribus hardware, and 
maintaining Pluribus systems. On the other hand, the set of 
documents is organized so that any one aspect of Pluribus 
endeavor (e.g., Pluribus manufacture) should be documented with 
a subset of the documents; thus, not everyone need carry all 
documents with him at all times--only those he needs. 


The chart on the following page suggests which Pluribus 
documents will be useful for which areas of endeavor and for. 


what types of people. 


FOR (AREA OF ENDEAVOR) : 


Designing an application 


system based on the Pluribus........ Ay 
Building a Pluribus Computer....... wo 
Using a Pluribus Computer........... 2, 


FOR (TYPE OF PERSON) : 


One considering buying a Pluribus...1,. 


A Pluribus factory worker...........7, 
A Pluribus hardware maintainer......2, 
-A-Pluribus programmer.........0.000+2; 


Table l: Guide to Pluribus Documents 


SEE DOCUMENT NUMBER: 


"ON WUoday 


—O86C. 


“OUT ueuMaN pue jauesag 2104 
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The documents have a loose-leaf format to facilitate 
updating. 


Of the nine, documents 1 through 6 will be available in 
reasonably large quantities. Documents 7, 8 and 9 contain much 
detail of little general interest (e.g., wire lists, assembly 
drawings) and are extremely cumbersome to produce; therefore, 
thelr availability from BBN will be quite limited, although 
they will be submitted to the National Technical Information 


service to allow general access. 


In the following paragraphs, we discuss each of the 
nine documents in turn, presenting the contents of each and 


discussing its expected use. 


Document 1: Overview. This document is meant to provide 
a quick summary of the Pluribus's capabilities, possible appli- 
cations, and architecture, and is the first document one should 


read to determine if he is at all interested in using a Pluribus. 


Doeument 2: System Handbook. This document is the primary 
text for one seeking familiarity with the Pluribus. The funda- 
mental ideas of the Pluribus are introduced and then discussed 
in detail, including the structure of the hardware and guidance 
on how we think the hardware should be configured and programmed. 
im particular, aiver @ Drie? .céneral description of the Pluribus 
system structure, there are discussions of the processor 
structure and of the addressing structure for the system, an 
outline of how programs might be written to use the Pluribus 
structure effectively, a discussion of Pluribus device handling 
and I/O handling, a discussion of the structure of the Pluribus 


busses and how they are coupled together, summaries of the 
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various devices which can be connected to the Pluribus, and a 
discussion of the Pluribus reliability mechanisms. While this 
document might best be thought of as a programmer's reference 
manual for the Pluribus, or alternatively, as the reference 

manual for Pluribus systems analysts, we think that everyone | 
associated with any phase of Pluribus development and use will 
be likely to want it, with the possible exception of those 


concerned with only very local aspects of Pluribus construction. 


This document is enhanced by the inclusion of a glossary, 
a guide to other documentation (which you are reading), an 
index, and some reprints of relevant papers written during the 
Pluribus development process which may give the reader greater 
insight into the use and structure of the Pluribus. 


Document 8: Conftgurator. This document lists the various 
components that make up Pluribus systems (e.¢., memories, 
processors, busses) and gives rules for configuring Pluribus 
systems. These rules are of two forms: rules of the first 
form are concerned with performance limitations; rules of the 
second form are concerned with physical limitations. An —_ 
example rule of the first form tells how effectively multiple 
processors on a bus can share a memory on the same bus as a 
function of processor speed and memory speed. An example rule 
of the second form says that if more than some number of cards 
are to pe used on a bus, then a bus extender will be needed. 
Of course, in some areas these two forms of rules are not 
independent; for instance, adding a bus extender may slow down 
the bus. | 


Report No. 2930 | Bolt Beranek and Newman Inc. 


This document will be used primarily by the systems 
analyst or system architect for a computer system using the 
Pluribus. Further, it will be necessary in order to price 
Pluribus systems accurately, since only careful configuration 


Will list all the system components actually needed. 


Document 4: Baste Software. This document presents only 
enough about the Piuribus software to enable the reader to 
program in basic machine language for the Pluribus. The Pluribus 
instruction set is presented, the several different Pluribus 
assembly languages are introduced, and there is a discussion 
of the basic debugging package which allows Pluribus memory 
locations and machine state information to be inspected and 


changed. 


Every Pluribus programmer will need to read this document 
as this is the software he will need to do "hands on" debugging 
of his program. Additionally, those building and maintaining 
Pluribus hardware systems will need to read this document 
because it describes the software they will need to operate 


hardware diagnostic programs. 


Document 5: Advanced Software. This document describes 
software beyond that needed just to debug programs and operate 
hardware diagnostics. The software available for the Lockheed 
SUE, the processor used in the Pluribus, is listed. Detailed 
descriptions and operating procedures are given for the two 


cross-assemblers available to assemble programs for the Pluribus. 
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The somewhat unstructured "package" that has been developed to 


permit reliable operation of the Pluribus is also discussed. 


Every Pluribus programmer, whether he is writing application 
programs, utility programs, or diagnostics, will need to refer 
to this document. 


Document 6: Funettonal Spectficattions. This document | 
provides the physical characteristics, operating characterisitcs, 
and necessary programming details for every Pluribus card. One 
way to think of this document is as an extension to Document 2, 


giving greater detail on specific devices. 


Document 7: Construetton. This document provides the 
information necessary to build the components of Pluribus 
systems. -For every Pliuribus card, the following are included: . 
parts list, wire list, art work, assembly drawing, and assembly 
procedure. For every mechanical part and cable used in a 
Pluribus, this document includes the following: parts list, 
assembly drawing, and assembly procedure. In addition, this 
document Contains @ section which inciudes. detalled instructions 


for any modifications and option selections for Pluribus cards. 


Document 8: Card Testing. This document gives instruction 
for testing every card that can be used in a Pluribus system. 
For Pluribus cards obtained from Lockheed, the Lockheed main- 
tenance bulletin and diagnostic procedure are provided. For 


ry card specially designed and constructed for the Pluribus, 
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this document includes the following: schematic diagrams, 


logic description, wire lists, test program, and test procedure. 


This document is necessary for anyone debugging cards, 
e1ther alver initial construction in..the Pluribus Tactory or 


after failure in the field. 


Document 9: System Integratton. This document describes 
how the components of a Pluribus system are assembled into 
a complete hardware system; e.g., how chassis mount in racks, 
how cards mount in chassis, and how to test the whole thing 
once it is together. Included in the document are an overview 
of the hardware system assembly process and the hardware system 
assembly procedure; option selection information; system test 
programs ; and finally, system quality control and acceptance 


porcedures. Tor newly constructed systems. 


This document is necessary for anyone debugging Pluribus 
SyStvens;.-61ther alter initial construction in the tactory or 


after failure in the field. 


A 
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Te INTRODUCTION 


The Pluribus* system is a general-purpose multiprocessor 
computer suitable for applications ranging from those normally 
identified with minicomputers to those typically associated with 
larger machines. Pluribus hardware has been designed so as to 
provide a suitable basis for the development of ultra-reliable 


hardware/software systems. 


Pluribus systems contain an arbitrary number of identical 
DPOCESSOrs each Of which has access both to its own Pravacve 
memory and to a common memory accessible by all processors. I/O 
devices which are part of the system can be controlled by any 
processor. The number of processors, size of common memory, and 


amount of I/O gear ona Pluribus system can be quite large. 


The Pluribus system achieves modularity and reliability by 
making all the processors equivalent. Any processor can perform 
any system task or control any device. Since each subsystem of 
the Pluribus system (processor, memory, and I/O) is expandable, 
systems can easily be configured to meet the throughput require- 
ments of a particular job. The scheme for interconnecting system 
components is also modular; hence, interconnection costs vary 


smoothly with system size. 


The Pluribus system was originally developed to serve as a 
modular reliable packet-switching node for the ARPA Network [1]. 
A node consisting of a 13-processor system is currently operational. 
The Pluribus aporoach 2s appropriate, however, for many other 
applications where reliability, modularity, or large logical com- 


puting power is required. 


Trademark of Bolt Beranek and Newman Ine. (BBN) 
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This handbook will describe the: structure and operation 
of the Pluribus system. It will emphasize utilization of the 
Pluribus architecture in the manner for which it was originally 
designed, although additional possibilities will become clear 
as the discussion progresses. The handbook is oriented both to 
the programmer who will use Le as. a basic reference document and 
to the system designer who will have to determine if the Pluribus 
is appropriate for his particular application. Section 2 
presents an overview of the Pluribus architecture. Section 3 
contains a brief description of the processor. Sections 4, 5, 
and 6 present the basic information concerning the Pluribus system; 
addressing, programming, and device handling. Section 7 discusses 
reliability machanisms in the Pluribus system, both hardware and 
software, in detail. Sections 8 and 9 discuss the Infibusses 
and bus couplers. Finally, Section 10 describes many device 
dependent: Teatures and oLts and: wilt be uUserul moss. Jikely tor 


reference purposes only. 


The Pluribus 2s Constructed out of components. Manuractured 
both by BBN and by Lockheed Electronics Company, Inc. (LEC). The 
BBN-produced hardware is described in detail in this document. 
LEC hardware from the SUE minicomputer. line is discussed only in 
sufficient detail to make the description of the Pluribus co-. 
herent. More complete information on the SUE components can be 
found in the Lockheed product literature [2,3,4,5]. 
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ee PLURIBUS SYSTEM STRUCTURE 


A Pluribus system consists of a number of components 
(processors, memory modules, and I/0 devices), a number of 
busses over which these components communicate, and a number 
of bus couplers which provide the mechanism for interconnecting 
the individual busses. Within this framework a wide variety 
of systems can be configured ranging from small single bus 
systems to large multi-bus systems with tens of processors, up 
to 1024K bytes of main memory, and a large assortment of I/0 
gear. The subsequent discussion will focus on a medium sized 
Pluribus: COnt terra on. Very small and wvwery Jarge. systems 
both involve additional considerations not discussed in detail 


here. 


The basic skeletal unit of the Pluribus system is the SUE 
Infibus* onto which all BBN and LEC devices are connected. The 
Infibus not only serves as a chassis into which device cards are 
plugged but also provides a means for communication among all 
attached devices. In general, a single Infibus can have an assort- 
ment of cards on it: processors, memories, or I/O devices. 
However, only one device can be in control of the Infibus at any 
eiven instant. An Infibus arbiter (Bus Control Unit: Card :or BCU) 
which must be present on the Infibus guarantees that this is the 
case. The total number of components which can be plugged onto a 
single Intibus. 2s. -dependent:0n the number, OF SOUS avaliable: 1or 
cards and the type of power supply used (see section 8.). 
Throughout the remainder of this document the word "bus" will be 


used as a shorthand for Infibus. 


*Trademark of Lockheed Electronics Company, Inc. 
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A small Pluribus system (even a single processor system) 
can be bUuLlD using only 4@ single bus. For many applications, 
however, the bandwidth capability and/or card capacity of a 
Single bus is exceeded and a multi-bus structure is required. 

In addition, applications which take advantage of the full capa- 
bilities of the Pluribus hardware for bandwidth and reliability 


will require multi-bus configurations. 


With more than one bus, the question becomes how To assign 
processors, memory, and I/O devices to the individual busses and 
how to connect them together. A typical configuration is illus- 
trated in Figure 1. Lines between busses represent bus couplers. 
Typically, busses in a Pluribus system are configured as one of 
three types: processor busses, memory busses, or I/O: busses. 
Processor busses support processors and private memory associated 
with each of the processors. Up to four processors (numbered 0-3) 
can logically be put ona single bus although contention for the 
bus. 2s Likely to reduce the effective. processor bandwidth. In the 
ARPA Network application, for example, four processors with con- 
tention produce the same computational capacity as would three 
processors if there were no interference among the processors (i.e., 
if the processors were actually independent). Although the con- 
tention is application-dependent, Pluribus systems will generally 
be configured with one, two, or three processors per processor bus. 
Two processors are indicated for the system illustrated in Figure 
1. The other components normally residing on the processor bus 
are the processor private memories. These memories will contain 
the "hot code" (i.e., those routines most frequently referenced) 
so as to reduce competition for the pool of shared (common) 
memory , and other code which is important to protect by 


removing it from shared memory. One useful technique 
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Figure 1] Typical Pluribus System Configuration 


Report No. 2930 Bolt Beranek and Newman Inc. 


is for all private memories to contain identical copies of the 
same code. Much of the system reliability software will be held 
in the private memories to guarantee that redundant copies exist 
in case of any memory faitlure. The maximum amount of private 
memory addressable by each processor is 16K bytes. Not shown in 
Figure 1 but existing on every bus is the BCU (bus arbiter) card. 
In certain cases it may be desirable to have some I/O devices on 
the processor bus, but this will be the cs cavuher (han the 


rule and is discussed further in section 6.4. 


Memory busses contain common memory shared by all processors. 
Up to 1008K bytes of common memory can be added in 8K or 16K byte 
increments. The ‘common memory will typically convain- ecde which 
is referenced less frequently than the "hot code". Generally, 
shared data structures, variables, and buffers will also be held 


in common memory. 


The configuration of common memory, that is, the assignment 
of memory modules to memory busses, depends on considerations of 
reliability and memory contention. For both reasons 1tU 18 desirable 
to have multiple memory modules on a bus, muitiple busses, and 
redundant copies of code and data structures. The detalls are 
application dependent and relate to the cost/performance (relia- 
bility) trade-offs which the system designer must: consider. For 
reliable operation at least two memory busses, two processor 


busses, and two I/O busses will be required. 


The I/O busses contain I/0 devices and the Pseudo Interrupt 
Device (PID) central to the Pluribus system operation. The PID 
keeps in hardware a list of what to do next. <A number can be 
written to the PID at any time and it will be remembered. When 


read, the PID returns (and deletes) the highest number it has 
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stored. By coding the numbers to represent tasks, and keeping the 
parameters of the tasks in memory, a processor can access the PID 
at the end of each task and determine very rapidly which task to do 
next. This approach is an important departure from the use of 
conventional interrupts and avoids the costs associated with saving 
and restoring machine state.* Further, this approach neatly side- 
SvepSsS The problem :-6f POouting 12nverrupts. to tne proper: processor: 


Wore Getvtalil on the tse of the PID 1s eiven im section 5. 


There can be no more than four PIDs in a Pluribus system. 
Even though some I/O busses may conceivably not contain a PID, or 


a bus may contain more than one, the usual configuration is one 
PID per I/O bus. 


In a Pluribus system, processor, memory, and I/O busses are 
connected by devices called bus couplers. The different types of 
DUS; COUDLers. required TO: Connect -GilTerent bis. palr types Toretner 
are GiSsCussed: tm more-delail an sectton 9s “Por moderate Sized 
systems, there will generally be a bus coupler between any two 
busses between which communication is required. Usually this 
iMpiLLeS <a. coupler Trom each bus cto all busses: of other Types. 
Thus the total number of bus couplers for such a Pluribus system 
with P processor busses, M memory busses, and I I/O busses is 
PM +M-I + P*-I. For smaller systems it is possible for one or 
more Infibusses to serve as combined memory and I/O busses, 
reducing the number of required bus couplers. For applications 
requiring large numbers of components (processors, memories and 
I/O devices) it will be possible to reduce the required number of 


bus couplers by building hierarchical Pliuribus systems where 


*Although not used within application software, conventional 
INLErPPUDUS: Can result. trom errors. and are used Tor special purooses. 


See SECELOn: 54:35 
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busses are not completely connected. A more detailed discussion 
of the issues and procedures for configuring a Pluribus system 


can be found in a separate document [6]. 


Several distinct processor models are available. The SUE 
is a relatively slow and inexpensive processor. Typical memory- 
to-register instructions have execution times on the order of 4 
microseconds. For a given application, the required processor 
power can be attained by using as many processors as are necessary. 
This approach to generating high throughput systems has the 
advantage of permitting extreme modularity and high reliability 


as well as graceful degradation. 
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ce PROCESSORS 
Suet Instruction Set and Format Summary 


Lockheed SUE processors are used as processor components 
for Pluribus systems. The basic processor is a microprogrammed 
general purpose 16-bit minicomputer with 8 general registers (one 
of these registers is the program counter), and a status and a 
control register. These registers (general purpose, status, and 
control) may be accessed externally by other devices via the 
IMtabus.. -In a. multiprocessor. this allows One processor TO -Ston 
sniStnen. examine sana: Chanpe. 1s repisters, and restart au. “Enere 
are 8 general instruction classes: MOVE, ADD, SUBTRACT, INCLUSIVE 
OR, EXCLUSIVE OR, AND, COMPARE, and TEST. Each of these instruc- 
tions can use a variety of addressing modes including register-to- 
PecisSter, MEMOry-LO=Pegister; resi sver—vo-memory . tndexeas an= 
CiPeCese. and auvooindexed. ASO aval laple are Trotlave,. shit ts; 
Conditional, branch, “UnmconGs tional. Dranch,. and. “suproucine Callas 
ATIStMUCTLONS.. ne -Cond1ci0ns vesved Tor branching are Divs 17 
the status register of the SUE. There are tests for result being 
ZeEVO., PESULE. DEINE Negatives «carry: On Last. arithmevic anstruceion, 
register value odd, overflow, value greater-than on last comparison, 
value equal on last comparison, and loop completion. The branch 
Can occur om either value: TRUE. or FALSE. J2nRStructions..are either 
one or two words long. The processor also contains 3 programmable 
flags, contained in the status register, that can be manipulated 
directly by instructions. The SUE processor recognizes and 
generates 16-bit addresses. In addition it contains a 2-bit KEY 
register which is settable by the SKEY instruction in the pro- 
cessor. The contents of this register are appended to the most 
significant end of the 16-bit address to generate an 18-bit address. 


Every memory access by a processor has these two bits appended. 
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Certain of these 18-bit addresses are mapped into 20-bit system 
addresses as described in section 4. The processor operates on 
either 16-bit words or 8-bit bytes of data. Bit @ is identified 
with the low order bit and bit 15 with the high order bit. Details 
of the SUE instruction set and the various processor types may be 


found in reference 4. 


re PROCESSOR STATES 

SUE Processors Cam be In one of Three states: “halted, 
running, or idle. Transitions between these states may be ef- 
fected either by the processor itself or by external manipulation. 


The implications of each of these three states are as follows: 


Ha Lt: NO INStruUCctsons: executed. InvErrupis Cisabted, 


registers externally accessible 


Run: INSTRUCTIONS executed, iInVeErrupes: enabled. 
| Pepisvers not. externally accessible except 


COnNTPOL PESISTEer « 


Idle: No instructions executed, interrupts enabled, 
registers not externally accessible except 


COntrol Treeister. 


External references to registers which are not accessible will 
result in -a@ QUIT, as described in sections 5.6.1 and 9.2. The 
Idle state is entered from the running state by executing a WAIT 
instruction; the HALT Svate,. by 2 HALT onstruction. The Run 
state is entered from the Idle state by the occurrence of an 
interrupt. External manipulation of these states operates as 


follows: 
The processor control register is the only register accessible 
while the processor is running. To halt a processor, a one is 


written to its control register. The processor will normally halt 
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When: Che ThstTructLon Le 18 Currencly executing 1s: completed. However. 
if the control register is read between the time that a zero 

is written to the control register and the time that the pro- 

CeSSOr completes 1S: current INnsStPuctLoOn., the falt signal wii 

be DOSt. ‘Hence, some -aleorithm such 2s. the following should 


De used TO guarantee that a processor coes 117 Tact. halv: 


Le ‘Weite I to the: processor Control Pesistver. 
Read some other processor register. 
If QUIT results go to L (see section 3.3). 
At. GhVs DOIG tne: Processor Ts dadved . 


LO Start a DrOocessor., one Must: Ani tialaze all aAnportany 
registers to needed values and store the number two LEGO Gae 
control register. This is done as follows: First it must 
be assured that the processor is halted. The program counter 
is initialized to the address of the program to be executed. 
The general registers are loaded with any values to be passed 
vO the program. Whe status register 1s initialized to specify 
the enabled interrupt levels, initial status flags, and pro- 
erammable flags. Bit 11 (hexadecimal constant 8882) should 
be set to activate the processor. Finally, writing a e to 
the control register starts the processor. An additional 
feature of the SUE processor is the ability to single step 
through an instruction sequence. The procedure fer doing 
tos: 12s: tdentical £6 That Of Stariing a processor excepL 


that a 3 is written to the control register rather than a ée. 
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3.3 QUIT Handling 


Processors requesting access to memory locations or I/0O 
FreCiLSCers dO: SO Dy Girectly or 1ndirectly placing the desired 
address on the appropriate bus along with the required operation 
(e.g. read, write) and any data required (for a write operation). 
When the requested operation is complete, the processor will 
receive a signal called DONE. If no device on the destination 
bus recognizes the address provided or if the device recognizing 
the address malfunctions, no DONE signal will be returned to 
the processor. Instead, after a fixed period of time, the bus 
arbiter on the requesting processor bus will send a QUIT signal 
CVO GAC PrPOCESSOr.,. Causing 2. COnVentional intverruot. (An equi- 


valent thing happens to requests by T/O devices from I/O busses.) 


In many applications a programmer will want to take some 
action based on the presence or absence of QUIT interrupts. In 
the ARPA Network app PLCatLons-Lor example, a2 device discovery 
routine in the reliability software searches system address Space 
and determines if known devices have disappeared or new ones 
appeared by attempting to read the devices! registers and 
eneckine Lor resulting QUT Tox. To provide this. mechanism che 
Inverrupy. Level POuUtTInNG whiten responds to QUITs-shoudd be wratiuen 


SO. that Conurol. Wiki be passed Dack To: The application prosram 
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if the application programmer has indicated that he wishes this 
to happen. He indicates this wish by surrounding the.instruc- 
tion which potentially causes the QUIT by an "unusual pattern" 
of other instructions. For example, if a programmer wants to 
check for a QUIT occurring when location ABC is referenced he 


might write the following: 


LDA A2, ABC 
NOP 
BR oa 
XL: QUIT BRANCH ADDRESS 


Ti no QUILTS Dproeram. COontLinues: here; 


The QUIT interrupt service, upon receiving COnCrOL.,. woud: cheek 
to see if the two instructions following the one which caused 
the QUIT were NOP and BR. +4. If they were, it would simply 
store the two bytes starting at location L (the address of the 
instruction causing the QUIT plus 8) in the program counter and 
dismiss the interrupt. If the two instructions at L-4 and 

L~2 do not match the NOP BR. +4 pattern, the interrupt service 
routine would take its usual action in handling the QUIT. Of 
course, references to ABC which do not cause QUITs cause 


execution to continue at Lte as indicated. 


\ 
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4. ADDRESSING 


A GCyptecal Pluribus conti teuration.1ncorporacine both 
private memory associated with each processor and a pool of 
common memory shared among all processors has been presented 
2. SCCG10n cy din thas Section the, Pauribus address Siruc ture 
is described in more detail. The application of this address 
Structure to: Pluribus: proeram structures will be discussed in 


section 5. 


In Pluribus systems all devices communicate with one 
another by writing into or reading from addresses. These ad- 
dresses may be memory locations, locations for controlling or 
interrogating I/O devices, or they may have some other special 
function. In any case it is important to understand two things; 
first, how addresses are generated and routed through the system 


and second, what things are referenced by what addresses. 


Addresses are generated by active devices, Chak: 2S GEevices 
wanting to read or write some location. This includes both 


processors and I/O devices. Consider first a processor. 


As indicated in Figure 3(a), SUE processors normally 
generate 16-bit addresses. With the addition of the 2-bit KEY 
register in the SUE, however, the Pluribus processors actually 
eenerate 18-bit addresses which are put on the processor bus. 
The KEY register can be changed under program control by 
execution of the SKEY instruction. For the class of applica- 
CLOns. DEeInNe Considered. 1n."vhis. document, Nowever., cach processor 
On 4 DUS: Will Gnitially set its KEY register Vo: 4a distinct 


value indicating its physical processor number and wiil not 
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PRIVATE MEM 
FOR PROC; 
O 
PROCESSOR. ADDRESSE 
a eee. * gree 
SOOS-3FFF - 
49OO-SFFF —— 
6DOO-7FFF ) REI ATIVE TO COMMON ADDRESS 
80Q0- 9FFF MAPS 1,2,3 SPACE 


AGO@- BFFF ? RESPECTIVELY SOODP 
CO@@- FBFF | 
FCQQ- FFFF 


PROCESSOR 

REGISTERS 

AND LOCAL 

|/O SPACE 

FCDOO | 
SYSTEM 
1/0 
. FFBFF SPACE 
Figure 2 Processor Address Space 
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KEY (2) | 18 BIT ADDRESS ON PROCESSOR BUS 
| ae Pa 
PROCESSOR ADDRESS (16) | 
Pe. 

(a) 
I8 BIT ADDRESS ON PROCESSOR BUS 14BIT ADDRESS RECOGNIZED BY 


PROCESSOR -XY'S PRIVATE MEMORY 


MAP REGISTER (7) 20 BIT ADDRESS PUT ON MEMORY BUS 
Pm ee: eee 
18 BIT ADDRESS ON PROCESSOR BUS | | 


iXYrst}oo re 
rs=Gl,I@ 
(c) 


20 BIT ADDRESS PUT ON 1/0 BUS 


I8 BIT ADDRESS ON PROCESSOR BUS 


tuvw# lill 
(d) 


18 BIT ADDRESS ON PROCESSOR BUS 


XV Ut 


ADDRESSES OF THIS FORM REFERENCE MAP REGISTERS, PROCESSOR BUS 1/0 
DEVICES, AUTO LOAD ROM, AND PROCESSOR AND CONSOLE REGISTERS 


(e) 


Figure 3 Address Mappings 
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normally chance thas secting. 


The KEY bits thus serve to differentiate the address spaces 
of the various (up to four) processors on the bus. whe Pi ghy= 
most bit of the 16 left to each processor is used to select left 
or right byte in byte mode instructions, allowing gis = 32K 
addressable words. in order to allow larger common memory and 
I/O space than this, provision has been made for mapping portions 
of this 32K processor address space onto a 512K word system ad- 


dress space. 


The addresses shown atthe left in Figure 2 are the 16-bit 
addresses generated by a typical processor (processor j). The 
manner in which the address space is accessed by any processor 
generated address depends on the range in which that address ilies. 
The four types of access will be discussed individually below. 

With all 4 types of access being discussed, the 18-bit address 
is simply put on the processor bus. Devices (memory, bus couplers, 
and I/O gear) able to recognize that address will respond and 


all others will ignore the address. 
4.1 References to Private Memory (QQ9QP9-3FFF): 


Any processor generated address in this range refers toa 
location. in the private memory associated with processor Jj on 
its processor bus. Up to 16K bytes of private memory can exist. 
As shown in Figure 3 (b), the high order two bits of the 18-bit 
address are used to select the proper memory module and the low 


order 14 bits select the location within the private memory. 
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4.2 References to Common Memory (4Q099-BFFF): 


Any processor generated address in this range refers to a 
location in common memory, that is, memory on one of the memory 
busses. There are 4 distinct sub-ranges within this range, each 
associated with a distinct hardware mapping register. This 
association 1s indicated in Pigure 2. Each. map register allows 
a contiguous set of 8K bytes of commom memory locations to be 
referenced. A separate set of 4 map registers is associated with 
each processor on a processor bus. The map registers are physi- 


cally located in the bus couplers (see section 9.1.) 


Figures 2 and 3(c) illustrate how this mapping into common 
memory is accomplished. An address in the range 4QQ0-SFFF 
implicitly refers to map register @. A 28-bit system address is 
developed at the processor end of the coupler by appending 7 bits 
from the map register to the low order 13-bits of the 18-bit 
address on the processor bus. This address is then forwarded to 


the common memory bus with the access request. 


Addresses in the range 6Q00-7FFF, 8000-9FFF, and AOOO-BFFF 
implicitly refer to map registers 1, 2, and 3 respectively and 
the identical type of mapping occurs when these sub-ranges are 
Pelerenced:. “The only special Tearure tn ohe way thar. Tae. Our 
maps work is related to memory read operations via map register 
3 which are transformed into read-modify-write accesses to 
common memory where the data rewritten is always zero. This 
allows the implementation of multiprocessor locks in the Pluribus 
system. More detail on the use of this feature is discussed in 


section 5.3 where Pluribus program structures are considered. 
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One final complication arises from the fact that a few of 
‘the first addresses on every memory and I/O bus are allocated for 
accessing the bus coupler control registers. The amount of this 
allocation depends on the number of couplers connected to the bus. 
In general, the memory words at these addresses should not be 
USeds GOP MOre delat tl: Om: “Vine Dus Coupler Control rep rsters, Sec 


section omer 


4.3 References to System I/0 Space (CPPP-FBFF): 


Any processor generated address in this range refers to a 
location in system I/O Space. In general, each Pluribus system 
device on an I/O bus appears to the processor as a set of 8 
contiguous registers (locations) in system I/O space. This block 
Oty PSP LSGers (1S Vet erred: 7O:-as tne Gevice Veoi ster Doers. F 
PPOCESSOr Cam acU1Vate a. device by writene conmands or data. vo 
certain (device dependent) addresses within the device register 
block. <A processor can interrogate a device by reading data or 
status registers within the 16 byte device register block. More 
detail about the allocation of system I/O space to multiple I/O 
bUSSes and about: Che sAnternal structure OF the device r,reeister 
block can be found in section 6 where device handling and I/O is 


discussed further. 


As indicated in Figure 3(d), the way that system I/O space 
addresses are developed is by appending four ones to the low-order 
16 bits of the 18-bit address on the processor bus. This is done 
automatically as is the appending of the map registers (discussed 


above) by hardware in the bus couplers. 


19 


Report No. 2930 ; Bolt Beranek and Newman Inc. 


4.4 References to Maps, Processor Registers, and Local I/0 Space 
(FCOO-FFFF): 


Any processor generated address in the range FCOO-FFFF 
(see Figure 3(e)) refers either to the map registers for that 
processor (in all the bus couplers attached to the processor 
bus) or refers to a part of the address space shared by all 
processors on a processor bus. The map registers must be ad- 
dressable, of course, so that a processor can dynamically modify 
the portions of the potentially large common memory to which it 
has access. Map registers 8, 1, 2, and 3 can be referenced via 
addresses FCOO, FCO2, FCO4, and FCO6 respectively. 


The local (to the processor bus) shared address space is 
assigned as shown in Figure 4. In general, I/O devices will be 
attached to an I/O Infibus in a Pluribus system. In some cases, 
however, it may be desirable or necessary to connect I/O devices 
directly on a processor bus. Addresses in the range FCO8 to 
FDFF will be used in such a configuration to refer to the device 
registers, similar to the way that the device register biock is 
used for referencing devices on the I/O Infibus. The auto load 

ROM is an optional hardware device attached to the processor bus 
which contains a program that when executed will cause a processor 
to load memory from a paper tape reader on the processor bus. 

The registers of all the processors and the processor bus console 
are accessible at addresses above FFOO. A processor should be 
halted before an actempt- FO read. any OF IS registers Occurs. 


Halting a processor is described in section 3. 
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PROCESSOR 0O 
REGISTERS 


PROCESSOR 1] 
REGISTERS 


PROCESSOR 2 
REGISTERS | 


PROCESSOR 3 
REGISTERS 


ADDRESS LIGHTS 


DATA LIGHTS | 


- UNASSIGNED 


pharroen on gewor” 
Se Te pone 
fd + a 
¢ 
1 
¥ 4 
4 


| REGISTER 1 
| REGISTER 3 
| REGISTER 5 
| REGISTER 6 
| REGISTER 7 | 


CONTROL REGISTER | |, 
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os PLURIBUS PROGRAM STRUCTURE 


In most current computer Systems a hardware priority 
interrupt mechanism is used to inform the program of the oc- 
currence of asynchronous external events. Since Pluribus systems 
do not generally use interrupts for this purpose, Pluribus pro- 
grams tend to be structured differently from programs developed 
for conventional machines. The fact that Pluribus programs are 
designed to operate in a multiprocessor environment imposes ad- 
ditional constraints on the. program structure. This section 
presents some of the issues and programming techniques which we 


believe are useful in developing Pluribus programs. 
5.1 Basic Control Structure: 


Before giving an example of a typical Pluribus program 
control structure, the basic operation of a PID will be reviewed 
(more detail on the PID can be found in section 10.1). The PID 
is a priority ordered memory device. It has a read address and 
a write address. When an even 86-bit number is Weitcen co 72 21D; 
the number is stored. When a PID is read, the largest 8-bit 
number stored in the PID will be returned and the number deleted 
from the PID. If nothing has been written to the PID, the read 
will return a value of zero. Numbers may be written to the PID 
both by hardware I/O devices and by software. Processors poll 
the PID for tasks to be executed. As a simple example of a | 
Pluribus control structure, consider asystem consisting of a 
number of Tasks which service 2 ‘set ot 1/70: devices. The ftol- 
lowing assembly language code could provide the framework for 


the required program. 
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TASKDISPATCHTABLE: MAINLOOP, TASK1, TASK 2, ..., TASKN 


MAINLOOP: LDA Al, PIDREADADDRESS 
JMP @ TASKDISPATCHTABLE (Al) 


TASK: 
JMP MAINLOOP 


The main loop of the program simply reads the PID and jumps to 
the appropriate task indirectly through TASKDISPATCHTABLE (i) 
where i is the value obtained by reading the PID. At the end 
of any task (e.g. TASKi), a jump to the main loop returns the 
processor to look for the next task to perform. If there is 
NOUnINe. 2m: the: Pip, Zero 2S returned -and The: processor Simply 
eyoles au, MAINLOOP.. -NOte that. 20 LS userud to. nave’ the PLD 
store even numbers only since the number retrieved will be used 


as an index into a table with two-byte entries. 


To allow tasks to be initiated by the software (e.g. TASKi 
to be initiated by TASKj), the following type of structure 


would be used: 
TASK]: 


LDA Al, TASKiPIDLEVEL 
STA Al, PIDWRITEADDRESS 


JMP MAINLOOP 
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5.2 System Response Time and Strips: 


As indicated in the above example, the way that I/O 
devices obtain service in a Pluribus system is to write the 
priority level of their service routine to the PID when they 
need attention, and wait for some processor to return to the 
main loop and pick up the associated task. Since the time 
that a device can wait for service before losing data may be 
critical, it 18 essential to configure systems and design soft- 


ware so that response time requirements can be met. 


The two main factors which influence the rate at which a 
Pluribus system can respond CO: Vee Dror Cy: ex vernal. Events are 
the total number of processors in the system and the duration 
of UVasSk Servicing tanstrucpion sequences: For <xample. 17 2 
Single processor system where the tasks are all of the form 
illustrated by the two previous examples, if the longest task 
execution time were T milliseconds, the maximum time which it 
could take to respond to an external event (i.e., notice that 
it had occurred) would also be T milliseconds. This worst case 
would happen only when the event occurred just after the single 
processor had picked up the longest task to run. Since in a 
Pluribus system there are no interrupts, the entire task cur- 
rently being executed runs to completion before there is 4 
reaction to the event (even though it may be of higher priority 


than the task currently being run). 


In the multiprocessor case, things are slightly more com- 
clicaved.  COMmstdering The: worst Case Pesponse. time as above, ft 
the ordered task execution times are Ty (smallest), T,; oe. 
Tn (largest) and there are P processors, the maximum time to 
Pesoond. vo0.an external event assuming n>p will be between 


ome, 


between ae and T depending on the number of incarnations of 
n | 


~ 
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a particular task which can exist simultaneously. Of course, 
the probability of such worst response times may be exceedingly 
small if the large tasks are run less frequently than the smalier 


tasks. 


Typical (average) rather than worst case response times 
wiil depend on three factors: (1) average task execution time, 


(2) number of processors P, and (3) average number of tasks, 


ee queued on the PID. Ji the average task execution Came 2s 

De and Nae Che typical time taken to service a high priority 
event will be Taye: if P>N, then there will usually be an idle 
processor which will immediately react to the external event and 


average response time will be essentially zero. 


In general, the application will dictate where strict 
real-time response must be guaranteed or if more flexible system 
response characteristics are adequate. If strict real-time 
response is required, then some program structure which permits 
both logical tasks of arbitrary length and fast response To 
critical external events may be required. To accomplish this, 
Pluribus: program tasks can be partitioned ianvo code segments 
referred to as strips. A strip is simply a sequence of instruc- 
tions within a task. A task can give up control of its proces- 
SOP sau sene> end: Or each Serio SO “Theat:.any higher priority tasks 
may be Pune Of course; if the task is incomplete at the ena of 
a strip, the task queues itself on the PID for further execution 
before yielding 268 processor. ‘The idea.26 2liustraved by che 


example below where TASKk is broken down into two strips. 
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DiSPATCHK: Kl | /INITIALIZE TO. THIS VALUE. 


TASKk: JMP @ DISPATCHk 


Kl: 
LDA Ae, = Ke 
STA Ae, DISPATCHkK 
| T 
LDA Al, TASKPIDLEVEL Strip 1 
STA Al, PIDWRITEADDRESS 
J MP MAINLOOP 
Ke: 


LDA A2, = Ki 
STA A2, DISPATCHk Siirip.2 
JMP MAINLOOP 


The first 1fistruetion of VASKK As a dispatch-To the segment 
(strip) of the code to be executed. This dispatch is initialized 
to Kl so when TASKk is first initiated, execution will begin at 
Kl. At the end of strip 1, the task stores a new dispatch ad- 
dress (K2) in the subtask dispatch location, DISPATCHK, writes 
its own PID level back into the PID and gives up the processor. 
The next time this PID level is serviced, the task will be re- 
sumed in strip 2 starting at Ke. At the end of Strip 2, the 
subtask dispatch location is restored so that strip 1 will be 
executed the next time that TASKk is activated. It must be kept 
in mind that .a task writing its own level to the PID will pre- 
vent the processor which is executing the task from picking up a 
Walling vask with lower prioracy. In Certain situations 10 may 
be desirable for a task to yield the processor and also "sleep" 
a specified period prior to getting rewritten to the PID. This 


can be accomplished by the task setting a software timer which 
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gets counted down by a periodic clock routine. When the timer 
reaches zero, the clock routine can write the sleeping task's 
level Do the PID. ‘The 5S ANnStructions at the end. of strip Lin 
Lhe. above ‘example miLeht,. Gherefore, be replaced: ‘by the Tollowing: 


LDA A2, = K2 

STA A2, DISPATCHk 
LDA A2, SLEEPTIME 
STA A2, TIMERk 
JMP MAINLOOP 


Then after TIMERk has been counted down, the timer routine will 


execute the instruction: 


LDA Al, TASKKPIDLEVEL 
STA Al, PIDWRITEADDRESS 


The decision of precisely where to segment a task into 
strips is somewhat arbitrary; the main rule is that the strips 
Mus. De -Shorv -enoush. So: Thal Che proper response Characveristi cs 
can be guaranteed. in the ARPA Network application of the 
Pluribus, for example, TL turned. Out. that. the proper typical 
strip size was on the order of 100 instructions (although a few 
infrequently run ones are much longer). As a rule of thumb, it 
will generally be sufficient to segment a task into strips as- 


suming each instruction takes 4 usec for execution. 


LwO: Other felatved: practical Dssues Prelevant. TO "Strip sLZe 
selections are convenience and overhead. In general, tasks should 
be broken into strips at convenient points in the code; that is, 
points at which little information (e.g. in the registers) needs 
LO be preserved. 10 May occasionally be desirable. to have surips 


somewhat smaller or larger than the nominal size so that such a 
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partitioning will be possible. Data which must be saved and re- 
stored across strip boundries adds to the already existing over- 
head associated with breaking the code into strips. In many 
applications 420, is likely that. dittie or no. breaking of Tasks 
into strips will be required. In the ARPA Network application, 
for example, multi-strip logical tasks are the exception rather 
than the rule. 


The fractional overhead associated with breaking a task into 
strips depends directly on the strip size since the number of 
instructions required for strip switching is essentially fixed. 
For example, in TASKk presented above 8 overhead instructions are 
associated with switching from one strip to another (6 in TASKk 
and 2 in the main loop). If the strip size were 100 instructions 
as is typically the case in the ARPANET application, then the 
processor overhead due to using strips would be 8%. In applica- 
tions where larger strips are acceptable, of course, the over- 
head- will be even smaller. Mxperience Wich 2. number -of Pluribus © 
system applications has indicated that the processor overhead 
and programmer effort associated with breaking tasks into strips 
is not a serious problem and 1S, a relatively small. price, 0. pay 
for the increased reliability and performance of the novel 


Pluribus architecture. 
5.3 Shared Data Structures, Shared Code, and Locks 


In a multiprocessor care must be exercised when a piece of 
data may be referenced (read and/or written) simultaneously by 
more than one processor. In this context, "simultaneously" 
means that a process running on one processor desires access to 
the data while another process running on a second processor 
already has access to the data. Consider, for example, two 


processors that are concurrently executing processes which 
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obtain buffers from a common free storage list. If some interlock 
is not used, it. ‘would be possibile for both processors to get the 
same buffer since the second processor could access the list after 
the first processor had accessed it but before the pointer was 
updated. 

To avoid this and a multitude of similar situations involving 
shared resources, a lock mechanism is typically used in programs 
for multiprocessors. Before a shared resource is accessed by a 
process, a logical lock is set. All processes determine if the 
lock is set prior to accessing the resource, and if so, then the 
process will wait. Only one process can, therefore, have access 
to the shared resource at any one time. 

To be effective it must be possible to test and set a lock in 
a single operation. A typical implementation provides the 
ability to read, test, and provisionally modify a memory location 
in a single interruptable operation. In Pluribus systems this 
feature is provided by turning memory reads through map register 3 
into read-modify-write accesses where the data rewritten is all 


ZEPOos. 


To implement a lock on a shared resource one simply assigns a 
location (LOCKVAR), addressed through map register 3, to the 
lock. The resource is unlocked if the lock is non-zero and 
locked otherwise. <A segment of code which accesses a locked 


resource might look as follows: 


Tacs LDA A2, LOCKVAR (hock-and continue). or (WAIT) 
BZ L 


access to shared resource 


Eke DUN. PC. GOCKVAR. + Unlock Lock 
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If a processor falls through the loop at L, the resource was 
unlocked but is now locked by the process running on this proces- 
sor. If a processor loops at L, then the shared resource is in 
use and the processor waits until the lock is released. To unlock 
the resource at Ll any non-zero quantity could have been stored in 
LOCKVAR. The current program counter (PC = general register 0) 
contains one such value which has the additional advantage of 
leaving a partial trace of the program execution in the lock 


registers. This trace may be helpful for debugging purposes. 


When a process encounters a locked resource, it may take one 
of two actions. As in the above example, it can remain in a tight 
loop checking the lock until it is unlocked. This type of waiting 
will be called busy waiting since the processor running the pro- 
cess remains occupied while waiting. Alternatively, a form of 
non-busy waiting may be implemented where the process may either 
write itself to the PID or set a timer so that a clock routine 
will later write it to the PID as described earlier. In either 
case the. processor then is free to seek other tasks while waiting. 
The busy form of a lock is appropriate in situations where the 
resource will be locked for only a short period. An example of 
this is the free buffer list accessing mentioned at the begin- 
ning of the discussion -on locks. The lock implementation which 
dispatches the processor to do other useful work will be more 
suitable in situations where the shared resource is likely to 
remain. docked tora relatavely jdong time. A paper vape reader 


shared by two processors might be such @ resource. 


The preceding discussion leaves considerable latitude with 
respect to what should be locked and when. For example, if each 
incarnation of a piece of shared code references a set of shared 


e 
variables, it may be more efficient to associate a single lock 
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with the set of shared variables than a lock with each of the 
individual variables. What needs to be balanced against this goal 
of fewer locks is the desire to keep locked segments short. 

Large locked segments, while reducing the total number of locking/ 
unlocking operations required,will tend to increase overhead due 
tO increased busy waiting or processor task switching. This 
overhead can become quite large on the percentage utilization of 
the shared resource increases beyond 60 = 70%. For this reason, 
the system designer must use considerable judgement in deciding 

on the extent of locked segments. In addition, locks should not 
remain locked across strip boundaries. Locked segments should 
aso De CxeCuved WU anLerrupes disabled so that prompt. unlocking 


of the shared resource is assured. 


One further consideration is that a processor may fail while 
executing a locked segment. ‘Two problems can arise in this case, 
(1) the locked resource will be unavailable to other tasks and 
(2) if busy waiting is implemented, processors may be executing 
infinite loops. Therefore, a processor should only be allowed To 
wait for the maximum amount of time which the lock can legiti- 
mately be set before deciding that a malfunction has occurred 


ang. .2CtTivatine @ recovery procedure: 


Cooperation with respect to the use of shared variables 
is required between tasks corresponding to different code 
segments and especially tasks corresponding to different incar- 
nations of the same reentrant code segment. In general, reentrant 
Coding Gs: partieularly “appropriave ina multiprocessor such as ‘ihe 
Piuribus system. The shared code may exist in common memory 
Or MULTID le COpLes: Of the Code may exist in the private processor 
memories to reduce contention. In the ARPA Network application, 
for example, shared code is used to transmit data from the IMP to 


each of a number of modems. In this case, the control structure 
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illustrated earlier in this section is modified to look as 


follows: 
TASKDISPATCHTABEL: MAINLOOP, TASK1, *** , MODEMOUT, MODEMOUT, «+ *TASKN 
CONTROLBLOCKS: @, BLOCK1, *** , MBLOCK1, MBLOCK2, *+** BLOCKN 


MAINLOOP: LDA Al, PIDREADADDRESS 
| JMP @TASKDISPATCHTABLE (Al) 


MODEMOUT: LDA A2@, CONTROLBLOCKS (Al) 
LDA A3, MODEMLOCK (A2) 


STA PC, MODEMLOCK 
JMP MAINLOOP 


The modem interfaces each write different levels to the PID when 
output of a buffer is complete but all these levels activate the 
seme “pilece of shared code, MODEMOUT. The PID levels are used; 
however, to select the address of a control block which contains 
the variables specific to the modem being serviced. At the start 
of MODEMOUT, an instruction is executed which loads an accumu- 
lator, (A2), with the address of this control block. One of the 
words in this biock. 16 a.loeck used to. lock ail. the other snared 
variables in the block. These variables remain locked for the 


duration of the modem output tasks. 
5.4 Using the Map Registers: 


The map registers allow four independent 8K byte segments of 
Che common memory to be referenced by each processor. The cnly 
constraint is that a read done through map register 3 will be a 


read and clear. The other three map registers may be used to 
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point. to program or data as required by the application. It is 

possible to have two map registers point to the same segment of 

memory. In the ARPA Network application, for example, map 3 

and one of the other map registers point to a segment containing 
system variables which can be accessed normally or used as lock. 


variables. 


In Pluribus systems with small memory configurations little 
of no map changing may be required. For applications requiring 
large primary memories, map changing will be more frequent. Of 
course, it is desirable to design a system so that as little map 
chaneing as possible will be.requiréed. ‘To-.change the areq.-o1 
common memory addressed through a particular map register, one 
Simply stores into the map register a constant whose high order 
{ bits are to become the contents of the mao. As already mentioned 
in section 4.4, the four maps have addresses FCO0, FCO02, FCO4, and 
FCO06. The code which changes a map must not itself be referenced 
through that map/ One way to make sure that this does not occur 


is to execute all map changing code out of private memory. 
9.9 Using Multiple PIDs 


The PID is the heart of the Pluribus system. Essentially all 
bask -dispatening 1s.<done via. this device. Lo 1s amportant,. therertore, 
that reliability provided by redundancy in the remainder of the Plu- 
ribUus: SyStem Components Nov De {ecKardi zed Dy availapliivy of only a 
single PID. | | 

LM. a Molva P LD system. the Pips. will themselves be Driority 
ordered, ‘Typically, the convurol: proeram in such a System: wii read 
tne Haehest. priority PID tarst.. Ita -PlpD other than the Lowest 
Oriority PLD returns wero, the Next tower priority FLD. well 


be read. If all PIDs return zero, the control program simply 
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eyecles by reading the highest priority PID again. 


As indicated earlier, a Pluribus system can have up to HPD oe 
one on each of 4 I/O busses. A hardware device on an I/O bus is | 
associated with a PID on that bus. Software tasks, on the other 
hand, may write to any of the PIDs in the system. Redundant I/0 
devices will generally be on different I/O busses and associated 


with different PIDs. 
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6. DEVICE HANDLING AND I[/0 


Pluribus systems may be comprised of two types of I/O 
devices, BBN-developed devices and Lockheed-developed devices. 
The primary distinctions between the two are that BBN devices 
interpret 20-bit addresses and use the PID while Lockheed devices 
interpret 16-bit addresses and utilize the standard SUE priority 
interrupt mechanism. Since SUE I/O programming is discussed at 
length in [2], most of this section will be devoted to the 
specifics of programming BBN-developed devices. Special con- 
siderations relevant to the programming of Lockheed I/O devices 


in a Pluribus environment are given at the end of the section. 


6.1 Address Structure 

As shown in Figure 2, system addresses FCOOO to FFBFF are 
reserved for Pluribus system I/O space. The detailed structure 
of this space depends on the allocation of addresses to I/0 
busses. Figure 5 shows one possible allocation of addresses in 
the case of a Pluribus with 2 I/O busses. Possible variations 


on this structure will be indicated later. 


The total system I/O space in Figure 5 is divided into four 
almost equal parts, two of which are assigned to each bus. The 
high address segment for each bus will be referred to as the 
primary I/O space and the low address segment as the auxiliary 
T/O space. Note that the primary address space of bus 1 (from 
address FFOOO to FFBFF) is shorter than the other 3 segments by 
1024 bytes because these 1024 addresses are allocated to 
Individual processor: maps. registers, and Local I/O space as 
shown in Figure 2. At the beginning of each primary address 
space are 144 bytes of reserved addresses. These locations are 
associated with the clock (RTC) and PID on the bus (see sections 


10.1 and 10.2), contain the bus coupler (BCM) control registers 
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(see section 9.2), and provide mapping for backwards bus: coupling 


(see section 7.1.3) using this bus. 


The remainder of the system I/O space is divided into 16-byte 
blocks where each block is associated with an I/O device (other 
than the clock and PID) attached to the bus. These blocks are 
called device register blocks. A processor activates an I/O 
device by writing to a certain address. within the device register 
block. <A processor can interrogate a device by reading the con- 
tents of status registers contained in this block. More detail 
on the structure of device register blocks is given below and is 
also contained in section 10. where individual I/O devices are 


discussed. 


Variation of the structure. shown in Figure 5 depends on the 
number of I/O busses and the allocation of system I/O addresses 
among them. This allocation is determined by switches on the bus 
couplers (see section 9.2). Figure 6 indicates allocations of 
system I/O space for 1, 2, 3, and 4 I/O busses. Only the primary 
I/O space allocations are shown; the auxiliary allocations are 
identical to these except that the highest address segment of 
auxiliary Ls (ne Same Size as the resv of the segments, thay 184 
it is not reduced in size by 1024 bytes. The low 144 bytes of 
each DFimMary SeemenL 2s Preserved On each bus as indicated an 
Posure 5. “While other allocations are possiple, tie: ones.-snAown 
in Figure 6 constitute all of the reasonable ones. Switch 
settings resulting in non-contiguous primary and auxiliary seg- 
ments for individual busses, while possible, are not considered 


here. 
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Figure 6 Allocations of Primary System I/0 Space 
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6.2 Programming BBN DMA (Direct Memory Access) I/0 Devices 


BBN DMA (Direct Memory Access) devices provide a means for the 
automatic transfer of blocks of data to (from) memory from (to) 
I/O devices on the I/O busses. While the DMA hardware and | 
its associated device interface are on separate Cargs.: 270m “one 


programmer's viewpoint they may be thought of as a single unit. 


In general, each data transfer will involve sending or re- 
ceiving a number of data buffers. Each data buffer will consist 
Of an ANVesrad. number of “words. Hor cach direction ‘or data flow 
(read, write) there are three main registers used by the programmer 
to control 1/0 operations; the begin memory (buffer) address 
register, the end memory (buffer) address register, and the status 
register. These registers are contained in the 16-byte device 
ree Lecer blocks: The structure of the device repister blocks Tor 
BBN DMA devices is shown in Figure 7. Each of these registers 


Le -GeecrLoed: nese Delow. 


DEVICE TYPE - The high order byte contains a number indicating 
the type of device interface involved (e.g. modem, host, etc.). 
This number is fixed by hardware in the device interface associa- 
ted with the DMA. In general, the low order byte contains the 
value set in the device number switches in the device interface. 
The device type register is readable; writing to it will have no 
effect. | 


RECEIVE/TRANSMIT BEGIN ADDRESS - These registers contain the 
high order 16 bits of the 20-bit system address specifying the 
first: Location. of the butfer to be read or writven. - Bits 1-3 

of the 20-bit starting address are contained in the receive or 
transmit status register (see below). Bit O of the 20-bit system 


address is always ®. The beginning address registers may be 


39 


Report No. 2930 © | Bolt Beranek and Newman Inc. 


either read or written. If read, the result returned is simply 
Zero. Normally when writing into this location, no data trans- 
mission will be in progress. in the direction correspondine to 

the register written (receive or transmit). The device will 
simply be initialized to transfer a buffer; actual data transfer 
does nov commence until the buffer end register is: written: 

If a transfer is in progress when the location is written, the 
transfer is aborted, the error bit (in the end address register 

~ see below) is set, the PID is written, and the corresponding 
half (receive or transmit ) of the device is initialized for trans-~ 


mission of a new buffer. 


RECEIVE/TRANSMIT END ADDRESS - These registers may be read or 
written. Normally, bits 0-12 of these registers will be written 
with the low order 13 bits of the address of the end of the 
burters (Bit 0 as ectualhly ignored and. assumed. ToO.be Zero.) 
WeLULNS “GO ‘This, aGdress tniviaves tne data transfer. <ATter the 
data transfer has ended, these registers can be read to determine 
information concerning the way that the transfer completed. | 

Bit: 155 20 Sel, INdleates Thal. no error has been devecied: and 
that this was the last buffer of the transfer. (Bit 15 will be 
set when the last buffer is transmitted correctly.) Bit O serves 
asan error Dit and -witl be set if: (1) the device was reini= 
tialized during the previous transmission (see above), (2) a QUIT 
occurred during transmission of the previous buffer, (3) the 
device is currently -active (see RECEIVE/TRANSMIT STATUS below) or 
(4) the device itself is reporting an error. Bits 1-12 of the 
end address register indicate the address, modulo Ber of: the 
last work actually transierred.. -[he Gop. 7 bits ol ‘Che: DMA polnter 
into the buffer come from the begin address (see above) and never 
change. Therefore, the buffer will "wrap around" on 8K byte 


boundaries in memory. 
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RECEIVE/TRANSMIT STATUS - The receive and transmit status regis- 
Vers May 2lso0. De DOLD Tread and wractten. “Wreavane che RESET bit 
causes the particular half of the interface (receive or transmit ) 
to reset itself. If that portion of the interface is active when 
Che reset is Lar Are the operation in progress will be aborted, 
the error bit in the end address register will be set. and the 
receive or transmit. level for the device will be written to the 
PID. Before initiating a DMA data transfer, bits 1-3 of the 
buffer beginning location must be written into bits 0-2 of the 
corresponding status register. Reading one of the status registers 
allows a processor to determine the PID level associated with 

that direction of data transfer and to interrogate the QUIT flag. 
The PID will be written and the QUIT flag will be set if a QUIT 
OCCUrred GUring the previous data transier pertormed Dy the. DMA. 
This: Could andicate @ parity error, non-existent. address,. etc. 

In this case, when the end pointer is read, the error bit will be 


set. 


The interpretation of the device dependent status bits varies 
from device to device but in general these bits provide for 
direct two-way communication between a processor and a device 


interface. 


One Ol The -Gevace Gevendent bits will be. the ACTIVE bit which. 
if set, indicates that a transfer GO OF TrOMm whe Cevyice a5 I 
progress. More precisely, a DMA device is active from the time 
that its end pointer is written (which starts the device) until 
the time that it writes its level to the PID Cind teat ine 2%. “is 


done). 


DEVICE DEPENDENT - This register can be optionally used by the 
device interface for any appropriate function. The assignment 


OF data. DitSs As arbitrary. 
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To cause a transfer to be performed by a BBN DMA Gevice. the 
program will typically perform the following steps: 
1. Write the STATUS REGISTER - This sets up the low-order 3 bits 
of the buffer start word address and selects any desired options 
(e.g., looped modem). This will normally be done only once for a 
sequence of DMA transfers. | 
2. Write the BEGIN ADDRESS REGISTER - This sets up the 16 high 
Orcer Divs OT the. DUT rer Stare. addqress. | 
3. Write the END ADDRESS REGISTER - This sets the end address of 
the buffer and initiates the DMA transfer. 
When ‘the PID level, andicatvine device completion... 18 picked up. Dy 
a. Processor’, it will: 
4. Read the END ADDRESS REGISTER and check bit 15 (completion). 
Le 2G. acs not set, the transfer has completed (i.e., no error 
occurred and this is the. last buffer of the transfer). Bits 1-12 
Are USEC “LO give the length or: the. bul fer. 
5. If this bit is not set, bit g (erpor) 28 (necked. df Dive 
Ie Zero. then? a1: “er ror ee bots DULLCr 2s MOG Tie, 2as 7 
Of the tvansrer. AS above, bits 1-12 are used to determine the 
Tener D: Or sone. bur Ter: 
os Uf Die w- 1s ‘one. then.an error has occurred. These are dif- 
ferentiated by examining the STATUS REGISTER. If bit 13 (active) 
is set, the device is still active and the PID value was spurious. 
If bit 8 (QUIT) iS: S60, @-QULT occurred during the transi er . 


Device dependent status bits may further define the error. 


In addition to the registers mentioned above, each BBN block 
transfer type of device has a number of manually settable switches. 
These switches, located on the device anvertace, re as follows 
(number of switches provided for each purpose shown in paren- 
theses): 


(i) Device Address Switch (10) - These switch settings define 
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the address of the device register block in I/O space (see 
Figure 5). The ten switches specify bits 413 of the address of 
the first register of the block. (Bits 14-19 of this address 


are all ones and bits 0-3 are all zeros.) 


(a) Receive/Transmit PID levels (7) - These seven switches 
define the number written to the PID upon completion of a data 
Leanster, ‘Hor duplex Gevices Chere are two: sets: OF Switches, 
For simplex devices only a single set is provided (e.g., CBT - 


see section 10.5). 


(iii): PID Address: (2) = Selects which of the 4 Pls: will “be 
written to by the device. The selected PID must be on the same 


Infibus as the device itself. 


(iv) Device Number (8) - In general, a set of 8 switches readable 
as the low-order byte of the first.word in the device register 
block (see Figure 7). The CBT device, however, has only one such 


Switch. 
6.3 BBN Non-DMA I/0 Devices 


Typically, non-DMA devices will only have a small amount of 
internal hardware buffering, therefore, Ley Need. TO: be Serviced py 
a processor no slower than every few byte times. The mechanism by 
Woiieh. SUCH “a “device 1S serviced can: take-one or two Torms ian Pluribus 
systems. One approach is to let the device be passive and put the 
responsibility tor servicing the device completely on. the processors. 
For input, the processors would have to DOLL .Gne Gevices faster than 
the input rate so that no data is lost. For output, Ene processors. 
would have to deliver data to the devices at a rate sufficient to 
guarantee that no undesirable gaps within the data occur. Although 
Such an approach permits a relatively simple hardware interface im- 
plementation, 20 may require an undesirable amount of processor 


overhead. 


4 3 


Report No. 2930 20° : Bolt Beranek and Newman Inc. 


111111 | 10-BIT DEVICE ADDRESS SWITCH | 0000 
WORD i—~ a 15 87 ® 


ERROR 
NO ERROR & LAST BUFFER OF 
TRANSFER (RECEIVE ONLY ) 


g | DEVICE INTERFACE | SPIT OF UGE NUMBER 
| TYPE H% 
DEVICE INTERFACE 
15 9 
P HIGH ORDER 16 BITS OF 
FIRST BUFFER LOCATION 
3 RECEIVE | 
STATUS 
4 TRANSMIT | | LAST BUFFER OF TRANSFER 
BEGIN ADDRESS — 15 ( TRANSMIT ONLY ) 12 
TRANSMIT | LOW ORDER 13 BITS 0] 
Pas ae: \{ WRITE) | OF BUFFER END ADDRESS * 
6 BITS 1-12 OF LAST 
| READ WORD TRANSFERRED ADDRESS 
7 __ DEVICE | | 
DEPENDENT 


WRITE 
DEVICE 


- -DEPENDENT-- 
STATUS 


READ 


* BIT © ON TRANSMIT END HAS A SPECIAL INTERPRETATION 
FOR THE CBT DEVICE (SEE SECTION 10.5) 


** ONLY ONE SWITCH EXISTS FOR CBT DEVICE (SEE SECTION 10.5) 


“Figure 7 DMA Registers 
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An alternate approach is to make the device active with respect 
to notifying the processors when it requires service. Ina Pluribus 
System this: implies Thar. the Gevice wail wrave its level to. the PLD 
when its internal buffers are ready. Checking whether the device 
needs service will, therefore, be done -autvomatically as part of 
toe. main: Pip reading loop of the program, ouch “an -approach, of 
course, requires more hardware in the device interface than does 


implementation of the first approach mentioned above. 


The only BBN non-DMA I/O device which currently exists is 
the synchronous line interface (SLI) which is described in detail 
in section 10./. TWhes Gevice 18 passive and consequently requires 
polling by the processors. Both DMA and non-DMA I/O devices which 
are addressed through system address space will have 16 byte device 
register DlOCcKS associated with them.: In contrast. to the DMA 
device register blocks which have a common format for all DMA 
devices, the structure of the non-DMA device register blocks will 


be device dependent. 
6.4 Lockheed SUE I/0 Devices 


As indicated above, standard SUE I/O devices differ from 
those developed specifically for the Pluribus system in. thay ee 
they interpret 16-bit addresses rather than 20-bit addresses and 
(2) if they are set up to actively notify the processor when they 
require. Service, they do. so Via-a hardware priority interrupt 
mechanism rather than via the PID. Since it will often be desirable 
to incorporate such devices inaPluribus system, some procedures 
for interfacing and programming them need to be developed. There 
are two distinct approaches that may be taken. First, sufficient 
modirieations could be made so that che device will. work -on vne 
system I/O busses. This approach has the advantage that the I/O 


device will be accessible to any processor in the system. It has 


45 


Report No. 2930 2 Bolt Beranek and Newman Inc. 


the disadvantage that hardware modifications probably need to be 
made to the device hardware. The other approach is simply to have 
the LEC device reside on one of the processor Infibusses, the place 
for which it was designed. This approach has the disadvantage of 
essentially. isolating the device from the processors in the system 
on other processor busses but has the advantage of requiring no 


hardware modifications. 


If the first approach is taken, -Unat. s.5 Che Gevice is puL.2on 
an I/O bus, the hardware modifications required depend on whether 
the device will be active or passive. in either case it will be 
essential to modify the device interface to recognize 20-bit ad- 
dresses. If it is to be a DMA-type device it would also be required 
to generate 20-bit addresses. In the ARPA Network applicavcion.. Lor 
example, a system control teletype has been interfaced in this manner 
and is handled by the processors as a passive device. Programming 
of LEC devices on the I/O busses will be similar to the BBN I/O 
devices discussed above. Hie CCval lS, OF "COURSE. "GeDend, On Ow: Uae 


device interfaces are modified. 


The “only Logical ditt iteulty with putting LEC. £/O devices. on 
the I/O busses arises in the case of high speed DMA devices which 
require fast servicing for proper operation.. To guarantee that 
such devices will be serviced within a specified time is likely to 
impose unacceptable constraints on the size of the strips into 
which tasks are partitioned: In such cases, e.g., handling a-disk,. 
it will probably be essential to take the second approach mentioned 


above and interface the device on one of the processor busses. 


Programming LEC I/O devices on a Pluribus processor bus is 
essentially identical to programming them in a standard multipro-— 
cessor SUE configuration. The only difference arises with DMA 


devices which are dealing with buffers in Pluribus common memory. 


46 


Report No. 2930 - Bolt Beranek and Newman Inc. 


Since the devices only produce 16-bit addresses, some mapping 
mechanism Similar to the processor address mapping 1s required 
here. This can be accomplished by simply dedicating one of the 
first three map registers associated with processor 0 to the I/O 
device for the duration of the time the device is being used. 
The I/O device addresses which have no key bits set appear to 
the bus couplers as requests from processor 0 and are mapped 
accordingly. I/O interrupts, when they occur, are always routed 
to processor 0. Even though it is undesirable from an overall 
system relrabilivy Standpoinl. this dependence on @ -speciric 


processor is unavoidable. 


More detail in programming specific LEC peripherals can be 
found in the LEC SUE Computer Handbook [2]. 
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7. SYSTEM RELIABILITY MECHANISMS 


ae hardware architecture of Pluribus systems which provides 
a foundation Lore tie. Ceve lopment of reliable computer systems has 
already been presented. This. section describes both some additional 
hardware mechanisms which have been included to improve system 
rellabilicy ‘and a general description of some software mechanisms 
which when operating on the Pluribus hardware can create a reliable 


computing environment in which LO @xecute apnolieatvion. prorcrans. 


The interpretation of reliability is strongly related to the 
Uype Of applications for whien<2 computer system is intended. At 
One -extreme: are computations which do not have strict real-time 
constraints, for example, large numerical computations. For these 
applications, reliability may mean simply checkpointing, that is, 
dumping intermediate states of the computation on a mass storage 
device so that the computation can be continued without much wasted 
effort should a system outage occur. At the other extreme are 
real-time control applications in which no outages are allowed and 
every Pequest. MUST De serviced within a. Short fixed Time: period. 
such applications may require simultaneously running the system on 
several identical hardware configurations with decisions based on 
amajority vote. Although the Pluribus system can be applied to 
applications in both of these two classes, the applications for 
which it was specifically designed fall Somewhere between these two. 
ihe Pluribus system will be most appropriate in situations where it 
is important to maximize MTBF and minimize MTTR but where occasional 


outages and minor delays in servicing requests can be tolerated. 


A reliable Pluribus system will generally be configured with 
at least one redundant copy of each hardware resource essential 
for running the overall system. It will be the goal of the relia- 


bility software to maintain a "working set" of resources and, if 
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possible, backup spares for each of them. In general, the relia- 
bility software will attempt to recover from failures of single 


hardware resources. 
7.1 Hardware Reliability Mechanisms 


The following mechanisms can be used. DOTH Dy The Pluribus 
reliability software described later in this section and applications 


programs which choose to use them directly. 
7.1.1) #=Power Failure/Restart Interrupts: 


Ca.) Processor Infibusses - The Infibus provides power for 
each of the devices it contains. If the power Supply 
is about to fail on a processor Infibus, processor 4 
on that bus receives a level 4 INVErTUPC. Wath, Geyice 
code 2. Processor £ then has approximately 2.5 milli- 
seconds to signal the other processors and save any 
important data on a non-failing Infibus or in non- 
destructive memory. When a processor Infibus is with- 
out power, the Control Register of any bus coupler con- 


necting thas: Inribus to another as. s¢i11..modirraple. 


When power is restored to the Infibus, a, PeSeL- Of Ghe 
Thatapus 2s -execuved sby The BCU and each device on the 
bus wild reinatialize itseit. Each processor will 
enter an idle state with all registers zeroed. Proces- 
sor @ will then execute a level 4 interrupt with 


device code 4 (power restart). 


ee Gece Memory and I/O Infibusses - When power is about to 
fail on a common Infibus, processor @ of each Infibus 
connected to the common Infibus executes a level l 


interrupt with device code 1. The processor must then 
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read the control register of each bus coupler on its 
Infibus to determine which one caused the interrupt. 
If the Control Register is @*, then the attached 


Infibus is losing power. 


When the processor determines which common Infibus 
is losing power, it has 2.5 milliseconds to signal 
other processors, save important data stored on the 
failing Infibus, and mark that Infibus unusable by 
the program. While the Infibus is without power the 


bus coupler map registers are still modifiable. 


LG Wii Oe Neeessary Lor Une processors: vO: Deriodi-= 
cally check the dead Iinfibus to see if power has been 
restored. When this Ls the case, Che Control Register 
will read (hexadecimal) 218%. 


Pic hoee Hardware Timeouts: 


A philosophy prevalent in the Pluribus hardware and 
software is that the system Should. Pperilorm suriiteiens 
introspection to recognize illegal and deadlocked states. 
If such states are detected, actions sufficient to move 
the system into some legal state should be initiated. 

The Infibus and device timeouts discussed below are two 


implementations of this general philosophy. 


*This may be modified by. kno: contents of any Ove eae ese memory 


location (see section 9. 2). 
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Vee Infibus Timeout - The Bus Control Unit monitors the 
frequency of activity on the Infibus. If there are no 
accesses for one second, .the BCU will execute an Infibus 
RESET and each device on the Infibus will reinitialize 
IGseli. ‘Processors on the bus. wild. enter -an idie: state 
Wied Lheie resisters SEG to Zero. Processor pw! sub- 
sequently receive a level 4 interrupt with device code 4 
power restart. 


Tere2ee Device Timeout and Multiple Interfaces - In many applica- 
tions of the Pluribus it will be important to have redun- 
dant (multiple) interfaces to one or more of the I/O 
CevViLeeS: 1 Order TO. aiprove Sysvem reliapliitys:. in. vie 
ARPA Network application, for example, multiple inter- 
faces to modems and Host COMpULErsS are planned. . Since 
such multiple interfaces will share a single I/O device, 
it will be necessary to electrically disable the device- 
interface transmit path on all interfaces but the one 


Currently am Use at. any eiven Time: 


Rather than have the enabling/disabling of these 
baths -cOontrolLled py Une processors. 'UNe: Anvertaces 
themselves are provided with sufficient logic so that 
the decision can be made locally. Each device interface 
which permit other interfaces like itself to be connected 
to a shared I/O device is equipped with a hardware timer. 
This timer is reset whenever a specific (device dependent ) 
word in the corresponding device register block is accessed. 
if the word is not accessed for a Fixed Lime. period, che 
LAmMer Piuns: CO: Zero: end. the~ess0Claved device: = 2ntvertace 
path is disabled. The path will be enabled whenever the 


Sspecalic word dn The device: register block Ss TesCG 
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referenced. A processor can, therefore, switch from 


one interface to a spare by simply stopping references to 


one interface and starting references to another. If for 


some reason an interface cannot be referenced, it will 


soon return to its stable, legal, disabled state. All 
DMA devices currently implement this facility and have 
one second timeouts. If a transfer is in progress when 


the interface timer reaches zero, the interface will 


short the transfer, write its level to the PID, and set 


the error bit in the associated device register block. 
The specific words in Che device regisver blocks: which 
must be referenced in order to reset the timers are given 
Bia) section 10, where the different ij O devices aré dis= 


cussed. 


7.1.3 Remote Reference/Control of Devices on a Processor Bus: 


Fil e3e) 


Backwards Bus Coupling - Using the type of interbus 
communication described so far, Lt: LS: OG. pOsSSsiDLe Lor 
a processor on one bus to interact with a processor on 
another bus except by voluntary communication on the 
part of both processors through a mutually agreed upon 
portion of shared memory. A processor cannot directly 
halts eStart. Or Modlry’ The regisvers.or local. memory 
of a processor on a different processor bus. To over- 
come this shortcoming, the Pluribus has an additional 
mechanism known as backwards bus coupling (BBC). Back- 
wards bus coupling permits requests to be transmitted 


to a processor bus via a bus coupler as well as from a 


* PROCeSsoOr bus; the normal direction. This is iiius-— 
trated in Figure 8a where a processor on bus A is trying 


e 
tO Verecrence some address local. to bus. B via I/O In- 
foie 204 
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Po} [Pr] [Ma] [Me Pa} LP] [Mo] [Me 


7 B 
ACNX NORMAL BACKWARDS,/Bc 
S, 


V4 


I/O BUS : 
Do 


(a) 


ADDRESS SPACE OF PROCESSOR WITH 
KEY BITS XY ON BUSB 


DIDO 
15 3210 
| ; 
kg 
MAP REGISTER 
ADDRESSES ee ss] \ EFFECTIVE 
ON BUS C b-------4 Py +H ]| ADDRESSES ON BUS B 
REFERENCED J (77~~7~—71 ppc [oa] I ACTUALLY REFERENCED 
BY PROCESSOR \ -_______ | MAPPING] ( BY PROCESSOR ON 
| : ee ed 
it eceuiece’ pO 
aes eae 
FFFF 
(b) 
Figure 8 Backwards Bus Coupling 
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Backwards bus coupling from one processor bus to- 
another. is only: possible over a shared I/O Infibus. In 
addition, only one bus coupler on an I/O bus can be enabled 
Lor backwards bus coupling at a time. Attempting to. 
enable more than one -BBC path on a Single bis. wid. produce 
unpredictable results. A lock will typically be as- 


sociated with this shared "BBC path’ resource. 


For backwards bus coupling to proceed, the BBC enable 
bit in the Control Register of the bus coupler connecting 
the shared I/O bus to the target processor bus must be 
set to one. (See Figure 5). This can be accomplished by 
writing the hexadecimal constant DET7D to. the Control 
register if forward coupling is to be disabled or DE7F if 
forward coupling is to be enabled. Tne Taes tj. DLS. Or 
these constants are a code word required to prevent indis- 
criminate modifications of the register by malfunctioning 
devices. | After the BBC. enable bit: is.set, the BBC Map 
Register (see. Figure oy) may be set to point to the desired 
area of address space on the target processor bus (at- 
Lempting £0 write the BBC map prior to enabling BBC will 
result in a QUIT). With BBC enabled and the map register 
set, reference to locations on the target processor bus 
may proceed by making reference Co corresponding bytes . 
within the BBC window (see Figure 5). After BBC accessing 
is complete, the coupler control register should be re- 
Cure: 16: 20s previous state by resetting the BBC enable 
Dit. | 


Figure 8b illustrates the details of the BBC map- 
ping in the context of the situation shown in Figure @a. 
To reference locations within the address space of the 


processor with key bits XY On DUS. By. processor A loads 


7 “64 


Report No. 2930 Bolt Beranek and Newman Inc. 


I 
is 


er 


bits 2 and 3 of the bus. C BBC. map register with XY and 
bits 3-15 of this map register with the high order 13 
address bits of an area on processor bus B. Subsequent 
references to bytes or words within the bus C BBC window 
are translated into 18-bit references on the target 


processor bus A at the address formed as follows: 


(low order 3-bits of 

XY Bits. 5=15.-0f Map: Register address of byte 
referenced in BBC 
Window) 


The BBC Map Register essentially serves as a base register 
which allows up to 8 bytes starting at the BBC map ad- 
aress v0. 'be rererenced.. OF course, che BEU: Man Rercister 
will need to be updated quite a bit if any significant 


number of BBC references are required. 


One Turt her complacavvon as the fact Chav simul 
taneous forward and backward bus coupling PEQUCSUS. COMA 4 
The result of such conflicts will be short term deadlocks 
While the Infibusses at both ends of the bus coupler 
time out their respective requests prior to sending QUITs 
to the requesting devices. In Figure Sa, for example, if 
processor Pe on bus A were. attempting to access memory M 
On (bigs: B sa - Che Same Time. Uhab. processor Po OF? [OU Ss. o> 
were attempting to access device D on I/O bus C, a 
Geadlock would.-o¢ccur Wath respect. To coupler Bo. Busses 
B and C would, therefore, CAimeouL the requests made to 
BC. DILor VO. .Senagrane QUITS. Go processor PS on bus B and 
bus: Coupler AG -~on bus. © Pespectively.. Since the time 
Untit a QUIT 1s returned. as “ypicaily Ponger Om 4a. proces= 


sor bus than on an I/0 bus; howéver,..uhe bus.-coupler AC 
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will Seva receive the Quit first and terminate the 
BBC request passing on the QUIT 5 the requesting proces- 
Sor. Py on bus. A. The forward bus coupling will then 


continue until completion, ‘The point of this discussion 


is that the application program using the BBC mechanism 
~ should be. aware that QUITS may result, be prepared to 


test for them (see section - 5. sy and repeat the BBC 


request 6a necessary. 


Lhe Soon tae list. summarizes the previous GLSCus si.0n: Wie a 


typical sequence of steps Co follow ror. BBC references: 


1) 
2) 
3) 
4) 


n~2) 
n~-1) 


Pe ere 


Lock (or wait on lock) BBC path resource On: £70 Dus 
Set BBC Enable bag. nie ‘Control. ue as 
Write Map Register | | | 
3 | —_ | : Will involve Scnsing and reacting 
Deu. 0. BBC sc to QUITs and may involve changes 
3 | tO map Desisver: 
Reset BBC Enable bit dn Contre? Register 
Unlock BBC path resource on I/O bus. 


Remote Resetting of a Processor Bus - Writing a zero to 


bit @ of the control register of a bus coupler connecting 


a shared (I/O or memory) bus to another processor bus 


will cause that processor Infibus to execute a RESET. 

AM devices on the processor bus will be reinivia lived: 
each processor will enter an idle state with all registers 
zeroed. A. subsequent 60° cycle clock interrupt (level 4, 
device code Ty will reactivate processor 4) on the bus. 

As with writing the BBC Enable Dats. Ne Tarsus Oe. “or. 
the word written to the control register must agree with 
ve hexadecimal constant DE78. Ty 2edqi1 tion, DIGS dang: 2 
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of the control register should be written 502s. VO vereave 
the proper state with respect to forward and backward 


coupling after the reset. 


PNase 3 Bus Amputation - Bus amputation provides a means of 
isolating selected active devices (1/0 devices and proces- 
sors) from the remainder of a Pluribus system. In this 
way malfunctioning devices can be prevented from af- 
fecting the heaithy system components. The unit of ampu- 
tation is the bus; that is, a whole bus must be amputated 


if any device on that bus is to be disabled. 


Bit 1 of the control register of a bus coupler must 
be at the coupler 38 -enabled for forward. DUS: cOouDline. 
By setting this bit @, therefore, all forward requests 
Over the coupler can be Diocked. By zeroing this bit in 
all bus couplers coming from a bus containing a malfunc- 
tLoning device, that device can .be removed from the 


Operational Piuribus. system. 


A -processor 28 not able to: write The control registers 
of any couplers connected to its bus (see section Oe ys 
therefore, amputation must be accomplished by references 
tO The Control resister erriving over a different. paul. 

In: Pieure. “o. -tor- example, coupler ac: could not be Shut 

off by Processor os on bus A. Processor Bs On DUS: Bs, 
however, Could Cut. path @C Wah. en “Access TO the ac 

control register (in the address space of bus C) via 
eoupler De... AS, WITH Wrivine the RESET and BBC Enable bats, 
the Lars, £5 Dsus Of Dae word weitven CO tne control 
register must agree with DE78 hexadecimal, therefore, the 
WOrG UO: Write“ TO The contro, register TO ampuUtave a4: DUS 

18 DET 20 BBG 2S: to. be disabled and DED. af BBC 1s: Go 

be. enapved. 
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PROCESSOR jp PROCESSOR B 
— BUS BUS 


MEMORY c. 
BUS 


cd 


Figure 9 Bus Amputation Example 
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To illustrate, suppose that in Figure 9 processor 
ee on bus B decided that processor P on bus A was mal- 
functioning and its bus should be amputated from the 
System, £0 Could Go This by simply zeroing bit. bin: the 
Control Tesgistver for couplers ec and ads Similariy., if 
processor Es determined that device X on I/O bus D was 
writing spurious data bits to common memory, it could 
LSOLaGe. tne Cevice by Zeroing bat 2 an the convrol 
register Tor coupler cd. This: would eilfectively remove 
bus D from the system as far as memory transfers are 
concerned although addresses on bus D could still be 


referenced via couplers ad and bd. 


7.1.4 Externally Initiated Reloads: 


For the ARPA Network application it was necessary to 
develop a means of reloading Piuribus systems remotely over 
phone lines. In other communication applications, Tne: ab Ld ty 


LO, do This may 2.50" be tmpoOrvant. 


A special piece of hardware called the Reload (RLD) 
Device is available which resides on an I/O bus and monitors 
up to 8 modem interfaces (receivers). When the RLD observes 
2.command in the input stream, it interprets the next 20 bits 
as a system address and the following 16 bits as a data word 
to be stored at that address. The address and data are heavily 
checksummed. Sequences of such commands can be sent to cause 
the RLD device to fill a portion of common memory or write via 
backwards bus coupling to the processor bus address spaces. 
Details eoncerning line protocol and device operation are 


given in section 10.6. 
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7.1.5 Parity Generation/Checking: 


Parity generation and checking schemes provide a simple and 
effective way to detect many of the errors which oceur in computer 
systems. The Pluribus uses such a scheme to automatically recognize 
memory failures and failures along the data transfer paths (i.e. bus 
couplers). This mechanism is invisible to the programmer except for 
the fact that a QUIT may result from a data access if bad parity is 
devecved. 


The type of parity calculated is called "address XOR Data" 
parity or AXD parity for short. AXD parity involves two parity 
bits for each 16-bit word, one associated with each 8-bit byte. 
Bach parity bit is calculated as the exclusive-or of the address 
parity and convents. parity of the byte. The advantage of this 
parity.function @s-that 10: decects: Cl) one data bit in. error, 
(2) all data bits zero, (3) all data bits one, and (4) one address 


DLO. A2n-error. 


There are essentially four distinct paths in a Pluribus system 
that implement parity checking. Each of these paths involves inter- 
bus transfers and is described below. Parity checking for data 


accesses on a Single bus is not implemented. 


(i) Processor/Common Memory Path - When data is being written 
to common memory the processor end of the bus coupler computes 
the AXD parity bits and sends them to be stored in the memory. 
On reading a memory location the stored parity bits are re- 
trieved and returned to the processor end of the bus coupler 
which recalculates the parity and matches it against the re- 
trieved bits. A QUIT will be eenerated if these two sets of 


Parity 0tts dO: nOe.. maten. 
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(ii) I/0 Device/Common Memory Path - The procedure here is 
virtually the same as for the Processor/Common Memory Path | 
above except that. the 170 end Or the: bus. coupler rather than 


the processor end checks the parity bits. 


(iii) Processor/I/0O Device Path - When a processor writes 
(reads) data to (from) one of the devices on an I/O bus, a 

Gif Terent. sort of parity checking is performed . A special 
device on the target I/O bus, the Parity (PAR) card, con- 
LINDOUS LY @enerates AXD: parity Trom eaddresses and data. placed 
on its bus during accesses to devices on that bus. This parity 
is fed back through the bus coupler involved in the reference 
and checked against parity computed at the processor end. A 
QUILT wads. be @eneratved 21 Ghe- two sets “of parity bits do. noe 
Match. This technique is relerred to as fPeedback-parity 


checking. 


(iv) Backwards Bus Coupling Path - Parity checking during BBC 
references is restricted to the forward part of the overall 
processor bus-to-processor bus path. The BBC registers are 
treated by the PAR card as if they were the registers of an 
I/O device on that bus, consequently the parity checking 


described in (iii) above applies. 


7.1.6 Transfers Between Private Memories On the Same Processor Bus 


Using the BBC mechanism it is possible for a processor on one 
processor bus to transfer data into the private memory of a proces- 
sor on another processor bus (see ection 7.1.3.1). It is also 
desirable for a processor to be able to effect transfers into the 
pravate memory of another processor on the same processor bus. 

Such transfers will be required, for example, when a processor on 


a bus wants to reload and restart another processing on that bus. 
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A recommended technique for doing this is described below. It 
involves running a short program out of the registers in one of 


the processors. 


Suppose processor O wishes to transfer N words from its private 
memory Starting at Jocation SOURCE: To consecutive words. starting 
at location DESTINATION in processor 1's private memory. Processor 
O first stores the following program in processor 1's registers 
(starting at FF20): 


LDA A2, SOURCE (-Al) 


KEY 1 

STA  A2, DESTINATION (Al) 

KEY O | 

BLP FF20 /Address of Processor 1 register @ 
JMP (AT) 


Next, Processor 0 sets up the count N in one of his registers (Al 


in the example above) by executing the following: 
LDA Al, =N 
Finally; processor O executes the program on processor 1's registers 
via a: 
JSB Ay, #E2O 
This example has assumed, of course, that processor 1 was iaieaia ty 


halted and that the original contents of processor 1's registers 


either did not matter or were initially saved and later restored. 


An attempt by the reader to work out some more straight forward 
solution should demonstrate the necessity of the sort of implemen- 


tation described above. 
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7.2 Software Reliability Mechanisms 


There are no -Strict. Constraints on the programmer ~oncerning 
how the Pluribus hardware features can be used. These hardware 
mechanisms have been developed, however, with a particular hardware/ 
software structure in mind. This structure will be described below. 
it should. be pointed out that there does not currently exist any 
reliability software package that is available with a Pluribus 
system. The Pluribus reliability software which now exists is 
integrated into the ARPA Network IMP application of the Pluribus. 
Nevervheless, we Deliceve that the basic: ideas embodied in this 
software (and perhaps much of the code itself) can be applied in 
other Pluribus: applications. and are, therefore, worvhwhile descriping 


here. 


One view of the relations between the three major. software 
components in 2: relaable Pluribus “system 2s shown ain Fagure (10. 
From the figure it can be seen that there are two major modules of 
the reliability software. The system reliability code is applica- 
tion independent and attempts to maintain a suitable set of re- 
sources in which to run the overall system. The appiication relia- 
DiLLeyY Code, On the ovher hand, 1s Govatly dependent Om the -parvicular 
appiTeatton. since atv has responsibility Por ¢cheeking and. Tixine 
bie “OAC Sovuecrures internal tO. Che A2pplI Cation. proerams fo 
develop this module one must have a detailed knowledge of the 
Staves: Of Vhat program. For this reason, the Polbowine discussion 


Wildl Toeus on the scructure of system reliabiiity code module. 


Under normal Carcums tances, Che: abo 1eCavion. vproecram wii be 
continuously running, executing application tasks fetched from the 
PID. The system timer routine which runs off of the real-time 
clock (RTC) causes both the application reliability code and the 


system reliability code to be. periodically executed. The system 
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reliability code is comprised of a sequence of stages that are 
performed when activated. These stages include such tasks as 
calculating “the cheeksum on programs in Pocal. and common memory . 
checking whether any memory or I/0 device has either appeared or 
disappeared, maintaining original and spare copies of code and 
variable segments, and maintaining the running status of all 
Processors: by reloading and. restarting “Uhem af necessavy. «it ali 
these tasks can be performed successfully, the system reliability 
software will return to the application program. This will normally 
be the case. In some situations, however, the system reliability 
code may be required to supervise the initialization of the applica- 
Lion: program Gtsell. -Relnivialization ‘Of tie: application. code, 
would be required, for example, if a segment of memory containing 
variables were taken out of service and a new portion of memory 


were allocated for this purpose. 


An important concept associated with the system reliability 
module is that of processor consensus. Before a processor is 
allowed to run either the application program or the application 
reliability code, it is necessary to establish a common environment 
for alt processors. This process of reaching an. agreement about 
the environment is called "forming a consensus", and we dub the 
group of agreeing processors "the Consensus". The work done by 
bie. Consensus ds ion Pact performed by individual processors, DuUL 
the coordination and discipline imposed on the Consensus members 
make them behave like a single logical entity. An example of a 
task requiring consensus is the identification of usable common 
memory and the assignment of functions (code, variables, buffers, 
ete.) to particular segments. The members of the Consensus may 
not agree in. their view of the environment, as tor example. when a 
broken bus coupler blinds one member to a segment of common memory. 


In this case the Consensus, including the processor with the broken 
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coupler, will agree to run the main system without that processor. 


In addition to periodic activation by the system timer routine, 
the system reliability code will also be activated following certain 
exceptional conditions indicated in Figure 10. Several of these 
conditions have already been discussed. An extremely important 
mechanism not yet mentioned, however, is the 60Hz interrupt which 
is used to guarantee that each processor does, in fact, periodically 
run the system reliability code. Hach processor upon executing the 
system reliability code sequence will reset a timer which the 60Hz 
IMCerrupu Service will count.downs i2f the timer ever reaches: Zero, 
a processor has been lax for one reason or another and the relia- 
bility code will try to get the processor running correctly again. 
As is the case for periodic activations, the system reliability 
code will eventually either go to sleep or supervise the initiali- 
Zation of the application reliability routine or the application 


DroOsram TUSeli « 


The discussion inthis section has only provided a: brier 
overview of the Pluribus software reliability mechanisms which are, 
Le tae ts. CUrrenLdy 2 Suete-OF Blues - More devas is: and @adarviona 
MOUCIVatTiION Tor Many Or the Gesien decisions réeleaving to Piuribus 


reliability mechanisms may be found in [7]. 
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APPLICATION DEPENDENT | APPLICATION INDEPENDENT 


, | 
PERIODIC ACTIVATION OF 
ALL ch cas PROCESSES 


CHECKS 
APPLICATION _ SYSTEM PERIODIC 
PROGRAM ii ah | ACTIVATION 


INITIALIZATION 


EXCEPTIONAL EVENTS: 


QUITs 

ILLEGAL OPERATIONS 

60 Hz INTERRUPT 

POWER FAIL INTERRUPT 
POWER RESTART INTERRUPT 


APPLICATION 
RELIABILITY 
CODE 


PERIODIC 
ACTIVATION 


Figure 10 Reliability Software 
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8. INFIBUSSES 


The Infibus is the primary power and communication path- 
way between devices. Physically, the bus is a panel containing 
24 slots. Each device is inserted into one or more of these slots. 
Power and signal circuits connect all of the Slots together. 
Power for the bus is provided by one of two possible power supplies: 
the smaller power supply is plugged into 8 of the slots of the bus, 
leaving 16 slots for devices; the larger power supply is external to 
the bus (leaving 24 slots for devices) and can provide power for 
one or more busses (depending on power requirements of the devices). 
The Bus Control Unit (BCU) module is necessary to control every 
IMtibuss ~Lt-OCCUDIES: One S16, Jeaving either 25: 0Fr.25 Silovs. tor 
Other devices. A bus can be extended by the addition of another 
DUS cabinets, The electronics for the extension wisi occupy one 
slot in each cabinet, leaving 29, 37, or 45 slots for devices. The 
number of slots occupied by the major components of the Pluribus 


system are as follows: 


Device Number of Slots 


Processor 
8k bytes Memory 
16k bytes Memory 
Bue: Control Unit (BCU 
Bus Coupler (BCP, BCM, or BCI - see section 9) 
PID Pseudo Interrupt Device 
RTC Real Time Clock 
HLC Local Host Interface 
CBT Checksum/Block Transfer 
ML Low Speed Modem Interface 
RLD Reload Card 
PAR Parity Module 
Sid SyRCnronows. bine Invertace 


PE HWM HRR HWW Pp 
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Electronically, the Infibus is the communications channel 
between devices. At any time, at. most one device contained ina 
bus: has access to that bus. ‘This device can request data from 
another device contained in the bus (read) or request that another 
GeViCe eee ye the data that this device is providing (write). 
‘The device which has access to the bus is called the bus Master. 
The Slave device is the device the bus Master is transferring data 
CO or from. The Master communicates with the other devices con- | 


tained in the tnribus by providing the following information: 


2O—-bit Address 
One of the control functions: 
Read 
Write 
Read-Modify-Write 
Whether data is word or byte 
Data (if function is Write) 


Parity 


Fach device contained in a bus continuously monitors the address 
being transmitted by the bus Master. A device becomes the Slave 
when it observes an address on the bus that it recognizes as its 
own. The device then performs the activity indicated by the ad- 
dress and control functions. When this activity is completed, a 
completion signal called DONE is returned. When the Master observes 
the DONE signal it accepts any data expected from the bus and 
relinquishes access to the bus: The Bus Control Unit. has.2au. thay 
time already chosen the next device to be Master from among the 
devices which have requested access to the bus but have not yet 
received it. It no- device recognizes the address that The Master 
provides the bus or S17. ‘vine Shave device maltuncltons:. Then nc 


action will be taken and no DONE signal will be returned. After 
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a fixed period of time (dependent on the particular bus), the 

Bus Control Unit will send a QUIT signal to the bus Master. The 
bus Master then relanguishes. control of the bus: and access 1s pro= 
Vided the next requesting device. The time allowed between access 
and a QUIT signal is established by the BCU hardware and is nor- 
mally between 5 and 5@@ microseconds. Processor busses will 
normally have the longest QUIT timeouts with I/O busses and memory 


busses having the next longest and shortest timeouts respectively. 


Iwo different devices on a bus can recognize the same address. 
If both of these devices respond with action and a DONE, the system 
will likely malfunction. Devices must, therefore, use some criteria 
external Go. the bus to resolve which device becomes the Slave. 
Normally the address recognition switches on each device in a sys- 
tem will be set to recognize disjoint portions of the system ad- 


QreEsSs: Space. 


Tie “Ours Provides: an. nite lizet1on) Siena. Ccatled ReSrl,. to 
each of the devices avtached to. 1G. This sienal 2s- transmitted vo 
the devices whenever power is being restored, whenever the bus is 
reset from the console or from another processor, or whenever there 
has been no transaction on this bus in the last second. Each 
device will terminate any activity when it receives the RESET Signal 


and reinitialize the state of all registers and indteavors. 
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a: Bus Couplers 


The functions of the bus couplers as components in an opera- 
tional Pluribus system have already been discussed in several 
earlier sections. In this section the internal structure of the 


bus couplers is considered in more detail. 


Haeh bus coupler connects two busses and, thererore,. has two 
ends. Each end of a bus coupler appears as a normal device on its 
containing Infibus. The 3 types of ends (BCP, BCM, and BCI) and 
two types of bus coupler (BCP-BCM and BCI-BCM) that may exist in 
a Pluribus Ssysvem: are itilustrated in: Figure Ji. BOP-BCM couplers are 
used to connect processor busses to either memory or I/O busses. 
BCI-BCM couplers are used to connect I/O busses to memory busses. 


The operation of the BCP, BOM, and BCI devices are presented below. 


9.1 BCP: | | 

Hach ‘BOP contains, Pour (=bit MAD Peg isvers ior each or. the 
POU POSSI Le processors On tne Imiibus. The MAP Peeisters sare 
numbered O-3 and are located in the address space OF eCach Processor 
at locations FCOZ-FC@6. Each processor has its own set of MAP 
registers, selected by bits 16 and 17 of the 18-bit address of 
data on the Infibus. These two bits are specified by the last 
execution of the SKEY instruction in the particular processor. 
The MAP registers can be modified by writing the new contents of 
the MAP to the corresponding address FC@@, FCOG2, FCOY, or FCKLE. 
The high order 7 bits of the word written become the new contents 
of - the map register. in general, bus coupler registers may be 
written but not read. Reading a MAP register gives a result of 
Zero and does not change the register. Infibus RESET does not 
effect the contents of the MAP register. The contents of the MAP 


registers are unpredictable at power-up. 
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I/O 


PROCESSOR = 
BUS BUS 


MEMORY 
BUS 


Figure 1] Types of Bus Couplers 
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During forward (normal) bus coupling the BCP is a Slave 
device on its bus and the BCP transforms each 18-bit processor 
address into a 20-bit system address to be sent to the BCM. As 
discussed in section 4., each processor's address space is divided 


up into 7 components: 


Addresses | Description 

OOSO-3FFF References to Local Memory 
4QOO-5SFFF Transform address using map 0 

60 00-7FFF | Transform address using map 1 
SOOG-IFFF Transform address using map 2 
AOOO-BFFF Transform address using map 3 
COOO-FBFF Transform address to I/O space 
FCQOO-FFFF References to Processor Registers 


and Local I/O space 


Addresses within @20@-3FFF or FC@@-FFF are ignored by the BCP. 

For those 8k byte segments of processor address space ah te 

a particular MAPS tae BCP forms a system address by preserving the 
low order 13 bits of the processor penerated address while replacing 
the hieh order 3. Divs by the 7 -bLt. contents Of The Corresponding 

MAP register. Addresses in the segment C@@@-FBFF are considered to 
be references to Pluribus device registers. The system address 

for such a reference consists of appending four bits of 1's to the 


JIMOST. Siem tacanc DOruzon of the address. 


When the BCP transforms an address, this address, any 

data, and one of the control operations (read, write, read-modify- 
write, byte) are communicated from the BCP to the attached BCM 
through 4 cable, Tne control operations willbe used by the BUM Toe 
generate a bus access on the target bus, generally identical to 
the bus access on the source bus except for the transformation of 


the address. Read operations using MAP 3, however, wiil be 
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transformed as previously described into read-modify-write accesses 
on the target bus (where the write data is zero) to allow imple- 


mentation of multiprocessor locks. 


When backwards bus coupling is enabled, the BCP acts as a 
Master on its bus and simply passes along the 18-bit references 
peneraved. Dy the BCM atu. whe -opner end of the cable. 


9.2 BCM: 

When a processor accesses a shared resource on a memory or 
T/O bus, all of the BCPs on the source bus map the initial address 
and pass it along to the BCM end of the bus coupler. Similarly, 
when an I/O device accesses a shared resource on a memory bus, 
aft “Une: BODS On Che Source ‘pus Transmit the 2nitial address. [0 
their BUM ena. Bach BOM then-Gerermines 2) tne address. sent TO 20 
LS One VO: Wher 126 can responds, Jf-1t 26 Nov, TAS BOM Simply 
ignores the request. If it is, the BCM requests access to its 
Infibus. When it receives control, the BCM transfers the 20-bit 
adCVESS, any Cava, and all cConvrol .setenals Go.1tts bus..and returns 
any. responses Pecelved to the Originating end: Or the bus eoupler 
pair. The addresses to which a BCM will respond are determined by 


the Cable Recognition Switch described below. 


There are two important reasons for making the bus coupler 
perform address discrimination. The first is to reduce hardware 
contention. If each BCM simply passed all addresses to the con- 
taining bus, every processor reference to common memory would be in 
contention for each memory bus rather than just the single bus on 
which the referenced memory was located. A similar contention 
problem would exist for processor references to I/O busses and I/O 
references to common memory. The second reason for BCM address 
QLScrimilnaclor is: Go eliminate multiple responses by the -connected 


busses. Since a bus always responds either positively (by DONE) or 
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negatively (by QUIT), one DONE and multiple QUITs would result from 
every access to common memory if no address discrimination was 

done. The QUITS would, of course, confuse the device that previous-— 
ly requested the access Since it would already have taken actions 
based. On: Lhe previous: DONE. “This same problem 1s the motivation 

for configuring Pluribus systems so that BCMs connected to different 
busses recognize disjoint areas of system address space. In general, 
The addresses recognized by all BCMs connected to the same bus will 


be identical. 


The BCM contains several physical switch registers which must 
be manually set and a single 16-bit control register which may be 
referenced under program control. The switch registers along with 


the number of bits (switches) in each register are indicated below: 


OWLE Ch Register Number of Bits 


MEMSW (Memory or I/O Bus) 


BCM CONTROL REGISTER ADDRESS 6 
BCM ADDRESS RECOGNITION: 
BASE 6 
RELEVANCE 


The algorithm used by the BCM for address discrimination is as 
TOllows; if the 20-bit agdress.1v. receives is. tess than COOL, 
then the high order 6 bits of the address are compared against 

the high order 6 bits in the BCM ADDRESS RECOGNITION switches. 

Ene GOmMparison LS. Savisriled Tor a particular address bit if either 
the corresponding RELEVANCE switch is OFF or the RELEVANCE switch 
is on and the address bit matches the corresponding switch (bit) in 
BASE. If all 6 high order bits satisfy the comparison, then the 


20-bit address is accepted and used to request a bus access. 
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Typically, the BASE AND RELEVANCE switches Will be set to recognize 
a contiguous portion of system address space. This is done by 
setting the high order 6 bits of BASE to some starting address and 
turning off some number of low order switches (within the high 
order 6) in RELEVANCE. Of course, more complicated memory access 
patterns can be admolemented by Other settings of the RELEVANCE 


Switches. 


If the 20-bit address passed to the BCM is greater than or 
equal to FCQZG and MEMSW is on, the address will not be recognized 
by the BCM. If the address is greater than or equal to FC#90 and 
MEMSW is off, bits 11 and 12 of the address must satisfy the com- 
parison Test deseribed: above With respect. to the two low order 
bits of the BCM ADDRESS RECOGNITION switches if the 2@-bit address 


is to be recognized and put on the (I/0) bus*. 


The BCM contains one internal register, the BCM Control 
Register. As already indicated in section 4.3 and 6.1, the block 
of addresses where these control registers can be found is at the 
beginning of one of the address space segments recognized by the 
bus to which the BCMs are connected. The precise location of a 
BCM Control Register is specified by the 6-bit BCM CONTROL REGISTER 
ADDRESS Switch. The number set in this switch is used as the dis- 
placement in words of the BCM Control Register from the starting 
address of the control register block. To state this more suc= 


CInctiy.. tne address. Of each. BCM contro. Pecister 1s: 


*Rarly models of the bus couplers required all 8 high order bits 
Of the address to match the. Switch bits Tor eddress- recognition 


CoO -OCeUr:. 
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Address Bits _ From | 

ie shom a High order 6 bits of BCM ADDRESS 
RECOGNITION BASE switches 

13 | | Negation of MEMSW | 

(cae, 0 

1-6 | 7 Contents of BCM CONTROL REGISTER 
ADDRESS switches. 

0 0 


The 3 switches (BBC Enable, Foreward Enable, and Reset) 
which can be set in the BCM COnUroOl FPESISter Dy 2 -processor have 
already been discussed in detail in section 7.1.3. It has also 
been pointed out that the data written to a BCM control register 
MUSt @aereée with the High order 13 Divs--of the nexadecimal constant 
DE78 if the write is to take effect. | 


One additional complexity in the BCM arises since the Control 
Registers for BCMs on a memory bus and those on an I/O bus will 
differ in one respect; those on a memory bus will share addresses 
with the memory devices on that bus whereas those on an I/O bus 
will not share locations with any 1/0 device. Since devices 
referencing’ BCM Control resisters wot expect a single DONE signal 
upon completion of an access, the BCM works as follows: if MEMSW 
is on, the BCM does not return a DONE since references to the 
control register also reference a memory location and the memory 
device returns the DONE signal. If MEMSW is off, on the other 
hand, the BCM generates and returns a DONE signal for references 
made to its control register since there is no other "overlapping 


devices" that will produce it. 
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The effect of two devices (BCM and memory) sharing the same 
address must also be kept in mind when the BCM control register is 
read or written. For BCMs attached to I/O busses, reading will 
return 2100 if the attached bus is up or 0 if the bus is in the 
process of going down due to a power failure-- see section 7.1.1 
(of course a QUIT will be returned if the attached bus is completely 
down). If the BCM shares the address of its control register with 
a Memory module, however, this 2lo0 or O will be Inciusive=-0r' ed 
with the contents of the associated memory word. For this reason 
and to permit proper operation OL The Pluribus, Sysvrem paericy 
mechanism, any read of a BCM control register will normally be 
preceded ‘by 2: write of zero To: the control regisver.. This wali 
clear any "shadow" memory location but not effect the control | 
register contents. The response to stores at the address of the BCM 
GCOnNTPOL Veri ster wid also depend on whether the address is on an 
I/O bus or a memory bus in addition to whether or not the high order 
13 bits of the data written match the key DE78. If the Key matches, 
the write will take effect and a DONE will be returned. If the Key 
does not match, a DONE response will be returned if the BCM control 
register is on a memory bus and a QUIT response will be returned if 


the BCM control register is on an I/O bus. 
Jes BOL: 


The BCI serves in place of a BCP when coupling an I/O Infibus 
CO a memory dInfibus. IlUs relation to the BCM 1s identical to that 
of the BCP with the following three exceptions: (1) no address 
mapping is performed (devices on I/O busses generate 28-bit addres- 
ses), (2) any address less than FC0@0 (with any data and control 
Signals) is passed directly through the BCI to the BCM, and (3) any 
address greater than or equal to FC@Q@ is ignored. Devices on an 


I/O Infibus cannot directly communicate, therefore, with I/O 
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devices on another I/0 Infibus any such communication must be done 
Via common memory. ‘The BCI-BCM bus coupler cannot be used for 


backwards bus coupling. 
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10. DEVICES 


In section 6, the general information necessary for program- 
ming both BBN-developed and LEC devices was discussed. This 
section provides additional device-dependent information for each 
of the BBN-developed devices. Similar information for Lockheed 


devices can be Tound-in the LEC Produet. Literature. 


10.1 Pseudo Interrupt Device (PID): 


The PID LS: a Priority memory device, The application of this 
OeVIES- CO The control of processes in -a Pluribus system has been 
CECSCr aoc at Length an section 5; A. Piuribus sSysvem may nave up co 
POUr andgepencdent. Pips... He. ancacared: am Nisure >. the ‘Pap and 
Real-Time Clock (RTC) share a 16-byte device register block in the 
address space of the containing I/0 bus. The three recisvers 1 
this block associated with the PID are the following: 


PID WRITE: 


When data sis Written into the: PID: WRITE resisvers, Divs 
1 through 7 of the data get a zero appended as bit 0 
and the resulting even 8-bit number is stored. Only 
one copy of any number is retained. When the PID WRITE 
PeelSLe” as: tead.. Che lareest- number stored 2m vhe: FLp 


is returned but not deleted. 
PLD READ: 


When the PID READ register is read, the largest number 
stored in the PID is returned and that number deleted 

from PID storage. If there iS no number stored in the 
PID, a zero is returned. An attempt to write the PID 

READ register will result ina QUIT. 
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PID CLEAR: 


When data is written into the PID CLEAR register, bits 

1 through 7 of the data set a zero appended as bit 0 and 
the resulting even 8-bit number is compared with the | 
memory of the PID. If that number is already in PID 
memory, it is deleted. When the PID CLEAR register is 
read, the dargest number an PLD memory 1s returned buv 
NOt -GeLe ved. | | 


The address of each PID is specified by a two bit switch 
on the device which selects bits li and I2 of: the device 
vecister ‘block starting address. Bits Q=10.0f the device 
register block starting address are zeros and bit 13-19 
are ones. The PID ecard has a set of lights which display 
the high order 7 bits of the largest number stored. 


10.2 Real-Time Clock (RTC): 


The Pluribus has two methods for timing events. Processor @ 
on each processor bus can recive an interrupt every 1/6@th of a 
second on processor interrupt level 4. This rate is too infrequent 
for many applications, however, and does not fit into Che proces-= 
sor independent structure of the Pluribus. For these reasons 
another timing device, the RIC, has been developed. The-RTC also 
‘provides a. common time reference for all the processors. The RTC 
provides aecess to a clock which is incremented every 100 micro= 
seconds and which generates two distinct PID levels periodically, 


one every 1.6 milliseconds and another every 25.6 milliseconds. 


As indicated in Fibure 5, the RTC and PID share a 16-byte 
device register block in the address space of the containing I/0 


Infibus. The five resisters in this: block associated wilh the RIC 
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are the following: 


CLOCK COUNTER: The 16-bit clock counter, The RTC increments 


this register every 100 microseconds. 


CLOCK PID LEVELS: -The high-order byte is the number which will 
be written to the PID every 25.6 ms. The low-order 
byte is the number which will be written to the PID 


every 1.6 ms. 


CLOCK READABLE REGISTER 1 - A switch-settable register. 
CLOCK READABLE REGISTER 2 - A switch-settable register. 
CLOCK READABLE REGISTER 3 - A switch-settable register. 


These five registers are all read-only. Attempting to write to 


them will have no effect. 


The RTC has a device timer which will stop the ciock if i? 
has not been accessed (i.e. none of its 5 registers have been read) 
within the past second. This allows the Infibus timeout mechanism 
LO detect ana recover from: the 2ilecal svave: where the. RIG -16 the 
only device putting requests on an T7O-Infibus.. The clock wit 
also stop on master (bus) reset. Reading any RTC register will 


start the clock again in these two cases. 


The address of each RTC is specified by a two-bit switch on 
the device which selects bits 11 and 12 of the device register — 
Dhock -Starcvine address. There is also a two-bit switch which 
specifies. The address of. the PIP To. be writven Toi The same way 
Normally, these two bits will be identical to the two bits which 
locate the RTC device register block. In any case, bits 0-10 of 
the device register block starting address are zeros and bits 13-19 
are ones. -Two seven—bit switches are also available on the RTC 


to specify the 1.6 second and 25.6 second PID levels. 
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10.3 Low Speed Modem Interface (ML): 


The ML connects the Pluribus to a 303 modem at speeds up to 
250kb. The ML will transmit or receive messages of an arbicrary 
even number of data bytes to or from the 383. The Pluribus need 
only specify a portion of the message to the ML at any time. This 
portion of a message is called a buffer and is delimited in core 
by two addresses provided to the ML by the Pluribus DMA. On 
transmission, the ML will transmit end of message characters if an 
end of message flag is associated with a buffer. On reception, a 
bulter wilt De read until Sitner the anpuv. area "provaded: 15° fail: 
or the end of message characters are detected. Three best options 
are available on the ML under program control: (1) the ML can be 

crosspatched to itself so that it takes its transmitted data back 
“in as receive data ignoring the 383 modem, (2) the ML can loop 

the 383 modem back thus testing both the ML and the 383 modem, and 
(3) the ML can be forced to send a zero checksum to test the error 
logic of the ML. oe 


To allow for multiple MLs connected bo the same modem, the 
device timeout feature described in section {.-l.2.2 has been im- 
plemented for the ML. Data buffering is provided on the ML card 
to tolerate delays in servicing of approximately 32 characters on 
DOth» LnNpUL. and OULDUL: Without LOSS Of data -or Jine utilization. 
Additional delays of indefinite length are tolerated on output by 


sending a line protocol idle sequence. 


All data on the communication line is organized as 8-bit 
bytes, and sent low-order bit first. There are four control 


bytes: 
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NAME CODE (Hexadecimal ) 
SYN 16 
DLE 10 
STX 02 
ETX | 83 


The protocol on the line looks as follows: 


Ooo a ied Dies a De® . 4-7 De Be iG Oe Sar: 


E E E 
Me es x L L xX ae 2 XY Be SE Ge Go ye 
NN E X T Hie a BON oie Oe Ab G23 N N 
j 
al 2 . 4 - 5 3 6 i, 
Notes: 
Ag At least two SYN characters separate adjacent messages. 


An idle line is filled with SYN characters. 


as The beginning of a message is indicated by the sequence DLE STX. 
The TO LrOwine -charaever As Texts Note thac ar the RL) card 
Ss USC, UAE LirsG DEG sot text must be zero if the message is 


not to be interpreted as a special reload message. 


Se The text is made up of characters taken from the buffer to be 
sent. The right half word is sent first, then the left hearts 
word. There are always an even number of text characters in 


a valid message, otherwise messages are of arbitrary length. 


Hs When the escape character DLE appears in the text, the hardware 


inserts an additional DLE. 


ae If text is not available to the ML in time, the sequence DLE SYN 


is sent as an idle protocol until text becomes available. 
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Gis The end of a packet is indicated by the sequence DLE ETX. 


‘o Each packet is followed by a 24-bit CRC checksum, sent as 3 
8-bit characters. The checksum is computed based on all of 
the characters in the message starting with DLE STX and 
ending with ETX. A bad checksum will cause the device 


receiver to report an error. 


Note that a DLE is always followed by a DLE, SYN, STX, or ETX. 

Any other character following a DLE will cause an error on receiving. 
An error may also be reported if the ML receiver is not serviced 
quickly enough, that is, within 64 character times of the time that. 
it writes the PID. <A receive reset command flushes the input buffer 
and forces the receive half of the interface to look for character 
syne. Detected errors also reset the interface to search for charac- 
ver sync, and tleae the dave as.end ol packer and ‘error, but donot 

f Lush: the but Ler. 


The ML is a DMA device and as such is programmed as described 
in section 6.2. That section also describes the switches and device 
independent bits in the ML (DMA) registers. The interpretations of 
device dependent bits in the registers are indicated below. In 


each case, ‘Uhe descripti1on- assumes. that a one 2s read Or wrivvens 
DEVICE. TYPES “Che Dben order DYte Contains “a. 1. 


RECEIVE. STATUS. (15) “loops 
Write: Cause the 383 modem to send back the transmission 
of the ML to the receive portion of the ML. The 
receive portion of the ML should be initialized 


before the transmit portion sO’ UhaU no date. 16 Ost. 


Read: The ML is looped. 
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RECHIVE STATUS. (14> "“Crosspatch": 
Write: Cause the ML to take back its transmitted data 
into the receive portion ignoring the 383 modem. 
The receive portion of the ML should be initialized 


before the transmit portion so that no data is lost. 


Read: The ML is cross-patched. 


RECEIVE. OUATUS (13). “hetive”. 
Read: The ML receive portion is either waiting for or 


transferring a buffer from the 343 modem to memory. 
TRANSMIT STATUS (15) "Device Timeout": 


Read: A one second timer has deactivated the Mb. If the 
Cransmit: Stavus word has not. D6en weitven For .one 
SeCCOnNds “ait 2ceavily Of Ghe Mi as: aborted. 
All ML circuits which communicate with the 33 
modem are deactivated. The ML will become usable 


again when the transmit status word is written. 


TRANSMIT STATUS (14) "Zero Checksum": | 
Write: Generate a zero checksum for this message. 


Reads A zero checksum will be generated for this message. 


TRANSMEY: STATUS (13). *heui ye": 
Read: A buffer is being transmitted. 
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10.4 Local Host Interface (HLC): 


The Local Host module provides an interface between the 
Pluribus and another computer (called a Host) according to the 
hardware specification for IMP to Host connections described in 
the BBN report, "Specifications for the Interconnection of a Host 
and an IMP" [8]. This is a general purpose asynchronous serial 
interiace,.. The Local Host module..can perform block transfers of 
data: 12m e@Lther direction between the Host:-and the PLUREBUS.. A 
-data block can be either read or written as a number of separate 
buffers if required. Transfer of the last buffer will have an 
associated end of data block flag. Padding is provided by the 
HUG. Veceiver av the end of data biocks To-accounG Lor word léengecn 
mismatch between the Pluribus and the attached Host. ‘Two padding 
options are available: (1) a1 followed by O's as described in 
Report No. 1822 or (2) all zeros. The choice is fixed by hardware 
Jumpers: On the t2nterface,, Another set or jumpers Perm.cvs. che 
Pluribus and Host ready lines to be permanently disabled (ignored). 
The Local Host module can be programmed to be looped. In this 
state, all data transmitted from the transmit half of the HLC is 
returned to the receive half of the HLC. This mode of operations 
is convenient for Hardware and software debugging. To allow for 
multiple HLCs connected -to the same Host, the device timeout feature 


described in section 7.1.2.2 has been implemented for the HLC. 


The HLC is a DMA device and as such is programmed as described 
in section 6.2. That section also describes the switches and 
device independent bits in the HLC (DMA) registers. The inter- 
pretations of device dependent bits in the registers are indicated 


below. In each case, the description assumes that a one is read 
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DEVICE TYPE: The high order byte contains a 2. 


RECEIVE STATUS (14): “Loop”: 

Wrate:. Connect the receive portion Or The: HLC tor the 
transmit portion so that data transmitted by the HLC will be re- 
turned to the receive portion of the HLC. To initiate the trans- 
mission both RECEIVE END AND TRANSMIT END must be written. RECEIVE 
END should be written before TRANSMIT END so that no data is lost. 
When the HLC is looped, the HOST READY indicator is the same as the 
Pluribus READY indicator. : 


Read: The HLC is performing in looped mode. 
RECEIVE SloATUS (Cla) “Actives 

Read: The receive portion of the HLC is active receiving 
a data block. 

RECEIVE SUATUS (©i2)- “Host: Ready: 

Read: A one indicates that the Host has set its ready 
Inageeavor:. 

RECEIVE STATUS.  (11)* 

Read: There was an error in the last buffer received from 
the Host. No end of message terminated the data block. This bit 
will be set if the Host Ready signal went away and returned during 
the previous transfer. Note that the error bit in the RECEIVE END 


PeESISteEr Wilt aso: be seu. 


RECHLVE. SLATUS. 1.0): | 
Read: Same as RECEIVE STATUS (11) above except if this bit 


is set, an end of message indication did terminate the data block. 
TRANSMIT STATUS (14) "Loop": 
Read: The HLC is performing in looped mode. 


TRANSMIE? STATUS “(i3) TRGGive vs . 
Read: The transmit portion of the HLC is active transmitting 
S. Cava DLOCKs | 
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TRANSMIT STATUS (12) "Pluribus Ready" 


Write: Writing a one sets the Pluribus ready indicator. 
Writing -a zero clears the Pluribus ready indicator. This indica- 
tor will also be cleared if neither the transmit Or FeCelvVe status 
words: has been written in the Jast second in: order to implement 
Coie: previously deseri bed GCevice Timeour. Deacure. 


Read: The Pluribus ready indicator LS: SU. 
10.5 Checksum/Block Transfer Device (CBT): 


The Checksum/Block Transfer Device performs one of three 
operations on a source data buffer: (1) it calculates a 16-bit 
checksum on the-data (2) it transmits the data words from the 
source buffer to a destination buffer, or (3) it does both (1) and 
(2) simultaneously. Aside from providing a convenient way to move 
data around Witla. Pluribus: SyYSvem.. Chis device provides a key 
service for the system reliability software (see section 7.2). To 
the DMA, the device appears as two separate sections - source 
(transmit) and destination (receive) which deal with the DMA data 
pranslrer independently but are linked closely together within the 
Ce vLee. Only CNe: SOuUrCEe InNTeErrupt and. status, however, are used. 
Checksum calculation is performed serially, low order bit first 
with DrPOVisSiTONn for elt ner reinitialiaZine Or  COntinuine. Ene compu 
tation when a new data block is specified. Either an IBM CRC 
16-bit checksum or a CCITT 16-bit checksum may be calculated. The 
choice is switch-selectable. The generator polynomiais for these 
checksums are as follows: IBM: ca + yr? + a a” He SEO (GS « 
yl& + xte + x? + be “TransrTer rate 225 limited only by bus access 
time when no checksum is being computed and by the slower of bus 
access or checksum computation when a checksum is being computed. 
Checksum computation time is approximately 1.3 microsecond per 
16-bit word. 
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The CBT is a DMA device and as such is programmed as described 
in section 6.2. That section also describes the switches and 
device independent bits in the CBT (DMA) registers. The interpreta- 
tions of device dependent bits in the registers are indicated below. 


In each case the description assumes that a one is read or written. 


DEVICE TYPE? “The hieh order byte .contains.@ 5. Sswiteh (bit) 0 
selects a CCITT checksum (off) or an IBM CRC16 checksum (on). 


TRANSMIT (SOURCE) END (15): 
Write: This is the last buffer of the block. 


TRANSMIT (SOURCE) END (0): 
WeLtTes Clear the checksum accumulavor register: Writing 
a zero indicates that the previous checksum should be pre- 
served, e.g. when checking a multi-buffer block. 


Read: HePor 


TRANSMIT (SOURCE) STATUS (15) "Check": | 
Write: Calculate checksum. Changing this bit while an 
operation iS in. progress will cause an anterrupt and a device 
VeSet. 


Reset: Checksum being calculated. 


TRANSMIT -<CSOURCE): STATUS: (la): “Granster"™: 
Write: Move data from source buffer to destination buffer. 
Writing this bit while an operation is in progress will 
cause an -interrupt and a device reset. 
Read: Data being moved from source buffer to destination 
butler. 


TRANSMIT -GSOURCE) STATUS -C13). "Aetive™: 


Read: CBT operation in progress. 
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TRANSMIT (SOURCE) SUATUS “CL2) “ROB: Destination": 
Read: Interrupt requested since the destination buffer 
is too small for source buffer. CBT has suspended activity 
until new destination buffer addresses are supplied for the 
remainder of data. .-No data is lost. The error bit is also 
set. 


TRANSMIT (SOURCE) STATUG CiL). “NOPY s 
Read: Interrupt requested since CBT initiated action but 
registers indicate that there is nothing to be done. The 


error bit is also set. 


TRANSIEEW SOURCE.) STATUS: (C0). “hast | | 
Read: TRANSMIT (SOURCE) END (15) was set when this buffer 


was written. 


TRANSMIT (SOURCE) STATUS: (9) “Destination QUIT": 
Read: The receive (destination) portion of the device 
received a QUIT during the previous operation. The error bit 


is also set. 


DEVICE DEPENDENT DMA REGISTER - The 16-bit checksum is accumulated 
here. This register may be initialized prior’ to the start of 24 
checksum computation. Writing this register during a check opera- 


tion will cause an erroneous checksum to be calculated. 


10.6 External Reload Device (RLD): 

The Reload card (RDD) monitors the input data from up to eight 
modem interfaces. When the RLD observes a command, it decodes the 
command as a 2f-bit system address, a 16-bit data word, and a 16-bit 
CRC16 checksum. This single card device is not processor control- 
lables “Out US ControLiabie by external sienals arravine Over che 
normal communication lines. The purpose of the RLD is to change. 


control, OF ‘Pestarc: Cie: i acuiei Dis system from a remote site without 
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on-site supervision. The RLD resides on an I/O bus, thus the RLD 
can modify common memory busses and access processor busses by 


backwards bus coupling as well as access devices on its own bus. 


Whenever a message is received over a communication line, 
the RLD checks the first bit (the least significant bit of the 
first 16-bit word). If this bit is one, the RLD determines that a 
sequence of RLD commands is arriving over that communication line 
and ceases to monitor the other 7 modem interfaces until all com- 
mands in the incoming message have been processed. Except for the 
first bit of the first command in the message, each of the remaining 
bits in each command are doubled to increase the uniqueness of the 
reload packet and to guarantee that the DLE character will not 
occur in the reload data stream. The format of an arriving RLD 


message is indicated below: 


AVAV21D DID ID/ C1 hc CID|E 
DID] |A|A|A/A/H/H|H H 
DIDI 1/TIT|T/TIEIEIE E}L|T 
R|R| |A/A/AjAlcicic C 
E/E|1 K|K K 
S|s S}s Sr | 
S|S}1 UU U| E| X 
M| M M 
12/16} | 0} 4] 8)12) 0 L2 
Wy} fet ya] 4} ¥ y 
Re he (me eos agate eas Le 


Tt should be noted That: 
1) The low-order two bits of the first word of the first command 


must: be: UL. The low order two bits of the first word of sub- 
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sequent commands in the message must be OO. 


2) An eight-bit padding byte follows the high order address bits. 
This byte can contain any non-zero "doubled" four-—-bit pattern. 
The pattern can be set by jumpers on the card. (Note that 


four ones are shown in the figure above.) 


3) An arbitrary number of commands may contained in an RLD 

message. | et 
4) The 16-bit checksum on the address and data bits 4n the IBM 
16 ‘ yi es 


CRC-16 checksum with generator polynomial X + xX> 4°10. 


For each command in the message, the RLD device stores the 
incoming data word at the specified system address. This process 
is repeated until either a bad checksum is detected, bad padding is 
detected, non-doubled data is detected, a delayed bus access occurs, 
or the RLD device times out after one second of inactivity. In 
each of these cases, the RLD device releases the communication 


line and is available to service one of the other modem interfaces. 


There are 3 lights on the RLD device which provide a visual 
indication of the device operation. One light is on from the time 
that the device is first activated until the bus containing the 
device is reset... The second 1s on for the duration of -@ single: hp 
eommand ‘sequences. “The Chird 1nagicavor 26 Drzerly dav Dy Compleulon 


of each bus access. 
10.7 Synchronous Line Interface (SLI): 


The SLI provides a simple synchronous full-duplex interface 


to a wide variety of modems. In contrast to the other interfaces 
previously described, the SLI is a single passive device and does 


not use either the DMA facility or the PID. To guarantee That 
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neither data not line bandwidth will be lost, the processors must 


poil each SLI in the system faster than the byte rate being used. 


Fach physical SLI card provides interfaces for two independent 
lines. The allocation of the 8 words in the device register block 
is given below. The location of the register block is fixed by 


jumpers on the card. 


Register 1: Device Type - Modem l 
Register 2: status - Modem 1] 
Register 3: Control - Modem 1 
Register 4: Data -—- Modem 1 
Register 5: Device Type - Modem 2 
Register 6: Status - Modem 2 
Register /: Control - Modem 2 
Register 8: Data - Modem 2 


The Device Type and Status words are read only registers whereas 
the Control and Data words are read-write registers. The inter- 
pretation of bits in each of these four registers are given below. 
In each case, the interpretation assumes that the particular bit is 


one unless otherwise stated. 


DEVICE TYPE (8-15): The high-order byte of the Device Type 


register contains a 4. 


DEVICE TYPE (6-7): These bits are set by switches on the SLI 
card and indicate information concerning the speed of the 


modem to which the SLI is connected. 


Bite: :C6,. 7) | Speed 
00 | Under 2.5K bits/sec 
O1 Under 5K bits/sec 
10 Under 10K bits/sec 
sie - -19.2K bits/sec and over. 
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STATUS (15): The transmitter buffer is empty. The next character 
may be written to the Data register. (It is assumed that 


Clear to Send status (10) - has previously been found to be on.) 


STATUS (14): A SYNC character has been transmitted. The program 
was too slow in responding to STATUS (15) above and in the 
absence of new data a SYNC character was transmitted. This 
Dit. remains: Set Until the next data character starts being 


transmitted. 


STATUS (10): Clear to Send. This is a signal received from the 
modem. In general, this indicator signifies that the modem: 
is ready to transmit data. Refer to the modem literature for 


more detail. 


STATUS (8): Data Set Ready. This is a signal received from the 
modem. In general, it indicates Lo tne Oi. that 26s modem 
is not in atest mode and its power is on. Refer to the 


modem literature for more detail. 


STATUS (COs/): Writer a Receiver Reset. (CONTROL bit’ 0) thas halt or 
the Status Register will monitor the input data stream, that 
LS, DLUS Wild. be deboured: here 2s well-as @o0ine Co: the: Data 
register, This wild continue until @ Zero has: propagated to 
STATUS O at which point these 8 bits will no longer change. 

A subsequent receiver reset will cause this first even char- 
acter search mode to start again. I2ItT 1s expected that this > 
feature will be used to handle the case of devices transmitting 
to the SLI which employ different syne characters. The first 
even character received can by mutual agreement be the syne 
character that will be recognized by the interface hardware 
(see DATA (8) and DATA (9) below). | 
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CONTROL: (2-0) Request to Send. This signal is passed directly 
to the modem. In general, it indicates to the modem that the 
SLI is ready to transmit data. The modem will normally 
respond by setting Clear to Send STATUS (10). Refer to the 


modem literature for more detail. 


CONTROL (9): Loop Test. Loop the SLI output back into the SLI 
input. This feature allows the SLI to test itself without a 


modem. 


CONTROL (7): Transmit and recieve in 7-bit plus parity mode. 

This bit will be set by the program when communicating with 
an ASCII terminal. When writing to the data register bits 

0-6 will be accepted and the SLI hardware will add the correct 
bit 7 to create odd parity in the 8~bit character transmitted. 
Data words received will be checked for odd parity (see DATA 
(9) below) but bit 7 of the data byte read will be zero. For 
communication with EBCDIC terminals, CONTROL (7) is cleared. 
In this case parity is neither generated nor checked. The 
6-bit character transmitted is the 8-bit byte written to the 


data register. 


CONTROL (0): Receiver Reset. Clear Received Parity Error - 
DATA (9), Receiver Overrun Error - DATA (8), Syne Received - 
DATA (14), and Data Ready - DATA (15). As described above, 
WreltUing this DLU @lso initiates syne character search mode and 
initializes STATUS (0-7) to all ones. 


in -contracy to tne DMA devices previously deseribed, the. input 
and output halves of the SLI share a single address for the two 
(read and write) DATA registers. The proper SLI internal register 


is referenced when the SLI is accessed as described below. 
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DATA (15) 


Read: Data Ready. Finding this bit set segnals that 
anew character is in bits 0-7 of the DATA register. 

If DATA (15) is zero, then no change has occurred to 

bits 0-7 of the DATA register since the last time the 
DATA register was read. Reading the DATA register 

sets bit 15 if it was one. In normal operation, the 

DATA register is read more frequently than the byte | 
rate, bit 15 is tested,and bits 0-7 extracted or ignored 


as appropriate. 
DATA CL) 


Read: Syne Received. This bit will be one from the 
time that a syne character is detected until a non-sync 
character is detected. Although available, this infor- 


maticn will generally not be used by most programs. 


DATA (9) 
Read: A parity error has been detected. This bit is 
cleared only by Receiver Reset. It is never set unless 
CONTROL (7) has been set to one. Parity checking is 
enabled when data mode is entered, that is, when the 
first non-syne character after two successive syne 


characters arrives. 


Write: Store Transmit Sync. Route DATA (0-7) to 4 
special holding register (rather than transmit it). 

This character wiil become the transmitted syne character, 
i.e. it will be transmitted whenever the last bit of 

a character has been sent but no new data character has 
been written to DATA (0-7). This register remains 


unchanged until rewritten. 
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DATA (8) | 
Read: The incoming data stream has not been serviced 
quickly enough and a character has been lost. Since 
the input is double buffered, two byte times must have 
elapsed since Data Ready was last set for this to occur. 


This bit is cleared only by Receiver Reset. 


Write: Store Receive Syne. Route DATA (0-7) to a 
second special holding register (rather than transmit 
it). This character will become the receive sync 
character, that is, this character will be compared to 
the received bit stream to achieve character 
synchronization. Data mode will be entered after at 
least two adjacent syne characters have been received. 


This register will remain unchanged until rewritten. 


DATA (0-7) 
Read: Input Data Byte. This is the data to be extracted 
when Data Ready is Set. 


Write: Output Data Byte. This is the location to which 
the next 8-bit byte should be written when Transmit 

Butter Empty is: Tound set... This. register 1s. not protected 
against premature writing and no indication is DEON eee 

if it is written when Transmit Buffer Empty is zero. 

If this happens, the character previously written will 
have been lost without being transmitted. Each receive 
and transmit portion of the SLI device is actually 

double buffered in addition to the serial-to-parallel 
shift register in the card. This extra buffering implies 
that the programmer actually has longer than a single 
character time in which to service the device. In:-eddi tion: 


the programmer should also be aware that this buffering 
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has other implications since the contents of the status 
and data registers are not synchronized. A status 
indicator can not be associated with the data byte 
currently available in the data register. 


The SLI device can be used with either switched or 
dedicated channels. The Data Set Ready, Data Terminal 
Ready, and Carrier Detect signals will be useful primarily 
in switched applications. They can all be strapped to 
"true" values for unswitched operation. If the SLI 

is used with a full-duplex channel (i.e. modem and 
circuit) the Request to Send and Clear to Send Signals 
could also be strapped "true". They are included to 


allow the option of half-~duplex operation. . 
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60 Hz. interrupt - a classical interrupt occurring at the 
power line frequency on level 4, device number 1l. 


abort - a QUIT. 


access time - time from the initiation of the request (rise 
of STRB) to the presentation or acknowledgment of data 
(rise of DONE). 


active - said of a DMA device while it is transferring data: 
from the writing of the end pointer to the setting of 
the PID level. 


active I/O device - an I/O device which indicates its need 
for service directly, usually either by classical or 
pseudo interrupt; cf. passive I/O device. 


address halt - a feature of the control panel which halts a 
processor when a_e selected address is accessed on the 
bus. 


address recognition - the process in which a module checks 
the Infibus address lines for an address which is in 
the range of those which pertain to that module. 


address space - the set of locations accessible to 
(addressable by) a device; cf. memory space, I/O 
space, system address space, processor address space, 
etc. 


amputate - to disconnect a bus (usually a processor. bus) 
from the rest of the system by turning off the forward 
enable bit in all bus couplers coming from that bus. 


ALD - a LEC card which implements the AutoLoaD function. 


arbitration - the act of choosing the next prospective user 
of a resource. 


ARPA Network - a national network of heterogeneous computers 
linked to facilitate research; the original design 
environment for the Pluribus. 


asynchronous - not necessarily occurring at a certain time 
or at fixed time intervals. 


asynchronous line - a serial communications line where the 
receiver derives timing information from the initial 
transition of a character's start bit; characters are 


sent individually, at arbitrary times, bounded by start © 


and stop bits. 
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attention - a classical interrupt on level 1, device number 
FF80, caused by pushing the "attn" button on the 
control panel. 7 | L_- + 


autoload - a LEC module which contains some read only memory 
programmed to do loading from any of a number of I/0 
devices; when commanded by a bus signal, the autoload 
will initiate a classical interrupt, having first set 
the vector address to point to the ROM. 


auto restart - see power recovery. 


auxiliary I/O space - the portion of I/O space from FC000 
through FDFFF. 


auxiliary processor - a processor which is not number 0 and 
thus does not handle classical interrupts. | 


AXD parity - a scheme wherein the parity bit(s) are derived 
from both the address and data; specifically, the 
parity bit of each byte is the exclusive-OR of the odd 
parity of the data and the odd parity of the byte 
address of that byte. 


backwards bus coupling - the process by which a master on a 
common (usually I/O or M/I) bus can access a slave on 
another bus (usually a processor bus); used by 
processors to access other processors' address space. 


bandwidth - the rate at which information may be transferred 
or processed. 


BBC - Backwards Bus Coupling. 


BBC enable bit - bit 2 of the bus coupler control register; 
controls whether that coupler is selected for BBC. 


BBC map - a register in the BCM which specifies the 
high-order address bits of a BBC reference. 


BBC window - the four word region of system address space 
through which BBC references are performed. | 


BBN - Bolt Beranek and Newman Inc.; the developer of 
Pluribus. 


BCI -— Bus Coupler I/O end; the card which forms the I/O end 
of an I/O-to-memory bus coupler. 


BCM - Bus Coupler Memory end; the card which forms the I/O 


end of a processor-to-I/O coupler and the memory end of 
processor-to-memory and I/O-to-memory bus couplers. 
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BCP - Bus Coupler Processor end; the card which forms’ the 
processor end of processor-to- memory and 
processor-to-I/0O bus couplers. 


BCU - Bus Control Unit; a LEC card which performs bus 
supervisory functions; it is chiefly responsible for 
arbitrating the use of the bus, but also assists in 
classical interrupts and other specialized functions. 


BDR - Bus Driver/Receiver: a custom IC used in both LEC and 
BBN boards to interface with the Infibus. 


begin pointer - in a DMA device, the address of the first 
word of the buffer. 


bezel - the decorative front of a bus unit which also 
contains an air filter. 


block transfer - the act of copying the contents of a series 
of contiguous memory locations to another place. 


buddy - the other processor(s) on the same bus. 


buffer - a series of contiguous memory locations which holds 
a block of data. 


bus - usually an abbreviation for Infibus. 


bus arbiter - a BCU. 
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bus controller - a BCU. 


bus coupler —- a module which allows transactions on one bus 
to be transformed into transactions on another bus, 
depending upon address; composed of a BCM, either a BCP 
or BCI, and connecting cables; performs other special 
features such as parity generation and checking, 
mapping, power isolation, and amputation. 


bus extender - a LEC module which allows one logical bus’ to 
span more than one bus unit; the extended bus looks 
just like one long bus; it consists of two cards, BxXD 
and BXR, and two connecting cables. 


bus timer - usually refers to the reset timer. 


bus unit - the basic mechanical module of the Pluribus; 
contains various combinations of Infibusses and power 
supplies, and has integral cooling. 


BXD —- Bus Extender Driver; the card which forms part of the 
bus extender; plugs into the same bus as the BCU. 
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BXR —- Bus Extender Receiver; the card which forms part of 
the bus extender; plugs into the bus which does not. 
have a BCU. : a 


byte - 8 bits; two bytes to a word. 


cable - an assembly which electrically connects two or more 
modules and/or external equipment; each type has a four 
letter designation. 


card - a logic board which plugs into the Infibus; each type 
has a three letter designation. 


CBT - a BBN card which forms part of a Checksum-Block 
Transfer module. 


CCITT checksum - a 16 bit checksum computed with polynomial 
RPSL G6 RPL et ees 


CCP - Communications and Control | Processor, a Pluribus 
application which involves the collection, limited 
processing and routing of seismic data. 


central processor - the number 0 processor electrically 
connected to the bus arbiter which handles all 
classical interrupts. 


checksum - a number of bits associated with a block of data 
computed via a fixed function from the data; the 
implicit redundancy can then be used to detect changes 
in the data. 


checksum-block transfer - a BBN module which allows’ the. 
computation of a cyclic checksum and/or the copying of 
a block of memory; consists of a DMA and a CBT. 


classical interrupt - the diversion. of the control stream of 
a processor in response to an external event; the 
device number of the interrupting device, status and 
program counter at the time of interruption are saved 
and the processor jumps indirect through a fixed 
location; also refers to the bus transaction which 
causes the interrupt. | | 


classical parity - the parity scheme wherein the source 
- generates on writes and the source checks on reads. 


clock - usually refers to RTC. 
CMB — the LEC printed circuit board which forms the actual 


Infibus; holds the edge connectors for the cards. 
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common bus - a bus which is not a processor bus: a memory, 
I/O, or M/I bus. 


common memory - that memory which can be accessed by all 
processors, that is, all memory on memory ot M/I 
busses. 


configuration - the process by which a group of Pluribus 
modules are selected and an arrangement designed to 
create a machine for a particular application. 


consensus - the agreement between processors that to take a 
particular action would be in their common interest; 
also refers to the process by which agreement is 
reached. 


console - usually refers to the control panel. 


contention - the situation where multiple users are 
attempting to simultaneously use a resource; this 
usually causes delay. 


continuous read/write - a feature of the control panel which 
when set, repeatedly performs the access requested by 
depressing the "read" or "write" buttons; located on 
the rear of the control panel. 


control panel - a LEC module which allows manual reading and 
writing of addresses, typically memory locations, 
processor registers, and starting and stopping of 
processors, and other special functions; consists of 
two cards, PCB and PBI, connected by a DIP connector 
cable, and a front panel, SWB, which connects to the 
other cards via three ribbon cables. 
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control register - a location associated with a module whose 
bits correspond to program settable functions; ina 
processor, register 15; in a serial or parallel 
interface, the address of the device + 6; in a bus 
coupler, set by the jumpers on the BCM. 


cooling module - the external shell of a bus unit which 
provides mechanical support for the Infibus chassis, 
fan pack, and bezel, and airflow isolation and 
deflection. 

CPA - a LEC card which forms part of the processor. 


CPB - an early LEC card which used to form part of the 
processor; superseded by CPC. 


CPC - a LEC card which forms part of the processor. 
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CPU - central processor; more generally, but incorrectly, a 
processor. , 


CRC-16 checksum - a 16 bit checksum ee rae ween porynoas 
x**16 + x*¥*15 + x**2 4 x**0. 


cycle eine - the time from the beginning of a request until 
the device has completed all activity related to that 
request and is ready to start or accept another; 
usually longer than access time. | 


cyclic checksum - a checksum computed by dividing the data 
by a specific polynomial and taking the remainder. 


D-cable - a cable which connects a card plugged into an 
Infibus with the fantail. 


DBAL -—- Dual Bus Access Logic, a custom IC that contains much 
of the logic necessary to be a bus master. 


DDT —- a program which allows the user to inspect and change 
memory locations and processor registers, start and 
stop processors, copy memory, field traps and other 
useful things; in many ways, can be thought of as a 
simple executive, providing an environment for user 
programs. 


deadlock - a state in which two (or more) processes 
(hardware or software) are each waiting for a resource 
held by the other; each now waits indefinitely for the 
other's resource to become available. 


device - usually a module that performs I/O functions; 
sometimes refers just to DMA devices. 


device dependent - a register or bit whose interpretation or 
function is determined by the particular module with 
which it is associated. 


device independent - a register or bit with a common 
interpretation or function over a range of different 
module types. | 


device register block - the eight word segment of address 
space which is associated with a DMA device. 


device type - a number indicating the type of the associated 
module; usually program readable in the low order byte 
of the first register of the device. 


device number - a number associated with each device which 
causes classical interrupts; when servicing an 
interrupt, this number can be read from the first word 
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of the interrupt vector, indicating which device caused 
the interrupt. 


DMA - Direct Memory Access; a BBN card which performs’ the 
bus interaction and pointer management for I/O devices. 


DMA device - an I/O device which uses a DMA; it transacts 
directly with memory with data in buffers. ; 


DONE - an MInfibus’ signal that indicates successful 
completion of a bus access cycle; also serves as the 
strobe for data in a read access. 


doubled cable - a cable which connects the two parts of a 
doubled interface with the fantail. 


doubled interface - an interface which, for reliability 
considerations, consists of two modules on different 
busses, connected such that either one can serve the 
external equipment. 


elastic buffer - a buffer which allows input and output to 
proceed asynchronously, at different rates. 


end pointer - in a DMA device, the address of the last word 
of a buffer. 


executive core - usually refers to locations 0-5E; the area 
in which interrupt, QUIT, and ILLOP information is 
stored. 
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EXY - Eight X and Y; a LEC card which forms part of the 
memory; contains the core stack itself. 


F-cable - a cable which connects external equipment to the 
fantail. 


fan pack - a chassis containing six fans that provide the 
cooling for each bus unit. 


fantail - a panel which contains connectors for cables from 
external equipment which interface to internal cables; 
used to facilitate the reconnection of external 
equipment. 


feedback parity - the parity scheme wherein the destination 
generates and the source checks parity on all 
transfers. 


flop - flip-flop. 


force reload —- a scheme by which memory locations may be 
loaded, processors started, etc., via special,, heavily 
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passworded messages on modem lines; used for remote 
start-up of machines. , 7 


forward enable bit - bit 1 of the bus coupler control 
register; when cleared, prevents all forward accesses 
through that coupler, thus amputating the bus 
associated with the BCP or BCI of that coupler. 


F-stick - to map an address in ‘T/o Space via the implicit 
fixed mapping. 


full duplex - a ccttuidaatien sath aes transmission can 
take place in both directions simultaneously. 


ground modem - not a satellite modem. 

half duplex - a communications path wherein transmission can 
take place in either direction, but not both 
simultaneously. 

halt(ed) - a state of the processor wherein instructions are 
not being executed, interrupts cannot be honored, and 
the registers are externally accessible. 

hex —- abbreviation for hexadecimal, base 16. 

high speed modem —- a BBN module which interfaces to a Bell 
306 modem at speeds up to 1. 5 Mbaud; consists of MHX, 
MHR, and DMA. 


HLC - Local Compatible Host; a BBN card that forms part of a 
Host interface. 


HIT - the name of the general Pluribus system test program. 


Host - a computer which provides and uses the actual network 
services; connected into the network via an IMP. 


Host interface -— a BBN module which interfaces to a local 
Host; comprised of a DMA and HLC. 


hot code — frequently executed code which is located in 
local memory. | 


IBM — four card interconnect module. 
IBM checksum - CRC-16 checksum. 
ICM —- three card interconnect module. 


IDM - two card interconnect module. 
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I-cable - a cable which connects two internal cards. 


idle - a state of the processor wherein no instructions are 
being executed, registers are not externally 
accessible, but interrupts may be honored. 


illegal operation - trap caused by attempted execution of an 
instruction not in the repertoire of the processor. 


ILLOP - illegal operation. 


ILLOP vector - the four word block holding information 
pertinent to the current ILLOP; starts at 20 for 
processor 0, 30 for processor 1; contents are: illegal 
instruction, status, program counter, address of 
service routine. 


IMP -— Interface Message Processor; the node computer of the 
ARPA Network, which performs the basic packet-switching 
functions. 


Infibus - the bus which physically and electrically connects 
the cards of a Pluribus system. 


interface - a module which allows access and information 
flow to and from external equipment. 


interrupt - usually a classical interrupt. 
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interrupt vector - the four word block holding information 
pertinent to a given interrupt level; for levels 1-4, 
starts at locations 0,8,10,18 respectively; contents 
are device number, status, program counter, address of 
service routine. 


I/O bus - a bus which contains primarily I/O devices. 


I/O space - the part of system address space from FC000 to 
FFBFF; the area which may be accessed via fixed mapping 
from processor address space; also refers to the 
corresponding section of processor address’ space 
(COOO0-FBFF) | 


isochronous line - a serial communications scheme wherein 
bit timing is derived from a separate clock line, but 
characters may be sent at arbitrary intervals and are 
bounded by start and poe oe ‘bits < 

jiffy - a 60 Hz. interrupt or 1/60th of a second. 


JIG - the name of the bus coupler stand-alone test program. 
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K - 1024 (decimal). 


key bits - address bits 16 and 17, asserted on processor 
references according to the contents of a two bit 
register set by the SKEY instruction; used to 
differentiate among the various processors on a bus. 


LEC - Lockheed Electronics Company. 
level 5 interrupt - an ILLOP. 
level 6 interrupt —- a QUIT. 


local Host - a Host interconnected via a bit serial, 
handshook interface, usually over distances of less 
than 30 feet. 


local memory - memory on a processor bus, as opposed to 
common memory. 


lock - a data structure (usually a single word) used to 
interlock processes; also refers to the act of reading 
a lock with a read-clear cycle. 


Lockheed Electronics Company - the manufacturer of several 
Pluribus parts. | 


low speed modem interface — a BBN module which interfaces to 
a Bell 303 modem at speeds up to 250 Kbaud; consists of 
a MLX, MLR, and a DMA. | 


map value - the 7 bit number that determines which 4K page 
of system address space is referred to by accesses in 
the associated map segment. 


map segment - one of the four 4K regions of processor 
address space through which accesses are made to common 
memory. | : 


mapping - the act of transforming an address in one address 
space to that in another. 


master - the participant in a bus’ transaction which 
initiates the access; e.g. the processor, when it is 
accessing memory. | | 

memory —- a LEC module, either 4K or 8K by either 16 or 18 
bits of random access core memory, consisting of three 
cards: TAG, SID, and EXY; also refers to a more generic © 
collection of the above. | 


memory bus —- a bus which contains primarily common memory. 
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memory space - the part of system address space from 0 to 
FBFFF. = 


message - the unit of data communicated between Hosts; 
messages are broken up by IMPs into one or more packets 
for transmission in the subnetwork. 


M/I bus - a common bus which contains both Memory and I/O. 


MHR - a BBN card which forms the Receive half of a 
High-speed ground Modem interface. 


MHX - a BBN card which forms the transmit half of a 
High-speed ground Modem interface. 


MLR -— a BBN card which forms the Receive half of a Low-speed 
ground Modem interface. 


MLX -— a BBN card which forms the transmit half of a 
Low-speed ground Modem interface. 


modem - a piece of external equipment which converts digital 
Signals from the computer to analog signals for 
communication and vice versa; also refers to modem 
interface. 


modem interface - the module which interfaces to a high 
speed synchronous modem, either ground or satellite, 
low or high speed. 
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module - a unit consisting of one or more cards’ which 
performs a unified function. 


MSR - the BBN card which forms the Receive part of the 
Satellite Modem interface. | 


MST - the BBN card which performs the Timing functions of 
the Satellite Modem interface. 


MSX — the BBN card which forms the transmit part of the 
Satellite Modem interface. 


multiprocessor - a system which contains several tightly 
coupled processors with some common resources. 


Multiwire - a technology for making cards, midway between 
printed circuit and wire wrap in the dimensions of cost 
and difficulty; consists of a printed circuit card 
which carries power and ground, covered by a sticky 
insulating layer, in which insulated wires are laid to 
form the signal paths. 
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P-cable - a cable which carries primarily power. 


packet - the unit of data communicated between IMPs on modem 
lines; several packets may form a message; ‘MouarEy on 
the order of one or two thousand Peco ; | 


packet-switching - a communications scheme in which packets 
of data from many sources are forwarded to many 
destinations along the same line, ee the use 
of the line at a high rate. ~ S 


page - a 4K region of common “memory, Or more generally, 
system address space. | 


PAR - I/0 PARity; a BBN card which etsreree parity for 
references to I/O devices. 


parallel interface -— a LEC module that can interface up to 
20 parallel bits of information; can be polled or use 
classical interrupts; used primarily as the paper tape 
reader interface; card type PPB. 


parity - the exclusive OR of a collection of data and /or 
address bits; also refers to schemes’ which detect 
changes by generating and later checking the parity of 
a collection of bits. , 


parity memory ~ memory which is 18 bits wide, allowing a 
parity bit to be stored for each byte. 7 


passive I/O device - a device which must be polled, does not 
interrupt; cf. active I/O device. 3 


password - a specific combination of data bits which must be 
written in order for an action to take place; used for 
reliability considerations. 


PBI - Panel Bus Interface; a LEC card which forms part of a 
control panel. 


PCB — Panel Control Board; a LEC card which forms part of a 
control panel. 


PCD - PreCeDence passer; a BBN card which serves only to 
pass the precedence pulse by an empty slot; used for 
debugging. | | ea 


PDU - Power Distribution Unit; usually refers to a BBN 
module which accepts site power and distributes it, 
with appropriate switches, circuit breakers and 
indicators; also refers to a LEC module which provides 
two key switches, one for power, the other for 
processor selection. 
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PID — Pseudo Interrupt Device, a BBN module which serves as 
a hardware pending task queue. 


PID level - the number that a device or a processor writes 
to the PID to signify that the associated task should 
be run. 


Pluribus _ a line of modular, reliable, 
multiprocessor/minicomputer systems produced by BBN. 


poll - the act of periodically checking a device to see if 
some event has occurred, as opposed to the device doing 
its own notification when a change in status occurs. 


power fail -— a classical interrupt which occurs 2.5 ms. 
before bus operations are ceased preparatory to 
complete power loss; occurs on level 4, device number 


2. 

power restart - a classical interrupt which occurs on 
restoration of local bus power on level 4, device 
number 4. 


power sense - an Infibus signal that indicates the condition 
of bus power, gives advance notice of a power failure; 
also refers to circuitry in the bus coupler that checks 
the status of power at each end of the coupler, 
allowing one end to disregard signals coming from a 
card with inadequate power. 
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power supply - a LEC module which supplies Infibus logic 
power and a 60 Hz. signal; comes in two styles: 
internal (plug-in, 5951) which takes up 8 of the 24 
slots of an Infibus, and external (stand-alone, 5952) 
which requires its own bus unit. 


PPB - Peripheral Parallel Buffer; parallel interface. 
precedence pulse - an Infibus signal which is daisy-chained 
between cards; used to resolve priority for the 


selection of the next bus master. 


primary I/O space - the portion of I/O space from FEQ000 to 
FFBFF. 


printed circuit - a technology for fabricating cards which 
involves etching away copper-clad epoxy boards to form 
the signal paths. | 


private memory - local memory. 


processor - a LEC module which executes instructions; 
consists of two cards, CPA and either CPB or CPC; three 
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microcode versions exist: standard, business, and 
scientific. | | 
processor address space - the address space seen by an 


individual processor; 32K words long. 


processor bus - a bus which contains processors and 
(usualiy) local. memory. 


processor bus address space - the aggregate of the four 
potential processor address spaces on a processor bus; 
128 K words long. 


pseudo interrupt - the act of writing a number to the PID to 
indicate that a task associated with that number should 
be performed. 


PSB - Peripheral Serial Buffer; serial interface. 


QUIT - an Infibus signal that indicates abnormal completion; 
e.g. non-existent device, malfunctioning device, 
parity error, etc.; also refers to the trap that is 
taken when a processor-initiated access results in a 
QUIT. : ; | | 


QUIT timer - the timer on the bus) arbiter which regulates 
how long the bus will wait for a DONE before deciding 
that the intended slave will not respond and thus 
should issue a QUIT, terminating the access; these 
timers have different values on different bus types. 


QUIT vector - the four word block of memory which records 
information pertinent to the most recent 
processor-~initiated QUIT; contents are: address 
referenced, status, program counter, and address of 
service routine; located at 28 for processor 0, at 38 

for processor l. 


rack - the unit which houses bus units; up to five bus 
units, a PDU, and a fantail may be mounted in one rack. 


read - an access in which data is transferred from slave to 
master. : | 


read-clear - a read-modify-write access in which the written 
data is zero. | 

read-modify-write - an access in which data is read from 
memory and then (potentially) different data is written 
back to the same address, all within one memory cycle. 


real time clock -— RTC. 
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reload card -—- RLD. 


remote power fail - a classical interrupt which indicates 
that a common bus's power is failing and about 2.5 ms. 
of usable power remains; occurs on level 1, device 
number 1. 


remote reset - the low-order bit of a bus coupler's' control 
register; when cleared, causes a reset to occur on the 
bus which the BCP is plugged into. 


reset timer —- see bus timer. 


resource -—- a part of a system needed by more than one of the 
parallel users and therefore a possible source of 
contention. 


ribbon cable - a multiconductor cable made of several 
parallel wires bonded together in a flat shape. 


run - a state of the processor in which instructions are 
being executed, interrupts may be honored, and 
registers are not externally accessible. 


ribbon - the part of an algorithm associated with a_e single 
PID level; may be one or more strips. 


RLD - ReLoaD; a BBN module which allows forced reloads. 
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round robin - a feature of the bus arbitration scheme which 
enforces fairness of access to the bus; when a device 
has been granted a bus access that terminates normally, 
it is not allowed to request another access until all 
those devices on that bus which are currently 
requesting access have been granted same. 


RTC - Real Time Clock; a BBN card which causes PID levels at 
intervals of 25.6 ms. and 1.6 ms., has a program 
readable 16 bit counter that increments each 100 wus., 
and three 16 bit readable switch registers. 


SACK timer - a timer on the bus arbiter which guards against 
the case wherein a potential bus master requests an 
access, subsequently stops’ the precedence pulse, 
indicating that it will be the next master, but fails 
to assert SACK, acknowledging this fact. 


satellite modem interface - a BBN module which interfaces to 
a satellite ground station transmitter; has features to 
enable use of the satellite channel in broadcast mode 
such as provision for switching the carrier and 
accurate timing of transmission and receipt of packets; 
consists of four boards: MSR, MST, MSX, and DMA. 
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scientific processor - the processor module with extended 
instruction set, including multiply, divide, double 
precision, etc. | —— | | 


select cycle - that part of an access which is concerned 
with selecting the next master. 


self interrupt - a trap. 


serial interface - a LEC module that interfaces asynchronous 
(start/stop) I/O devices; strappable for various 
speeds, character sizes, EIA vs. current loop, modem 
options, etc.; is half duplex and may either be polled 
or use classical interrupts; card type PSB. 


service cycle - that part of an access wherein the master 
actually transacts with the slave. 


SIMP - Satellite Interface Message Processor; an IMP which 
uses broadcast satellite channels as some of its 
communications links. 


simplex - a communications path wherein communication can 
take place in only one direction. 


slave - the participant in a bus transaction which responds 
to the master's request; e.g. the memory, when the 
processor is accessing it. 


SID - Sense and Inhibit Drivers - a LEC card which forms 
part of the memory. 


obi - = Synchronous Line Interface - a BBN card which 
interfaces two synchronou lines; a passive device 
which must be polled. 3 | | 


SMS - Synchronous Modem Simulator; a BBN card which 
interfaces two synchronous data sources, giving the 
appearance that the Pluribus is a modem; a passive 
device which must be polled. 


SRN - System Release Notice. 


status register - a location associated with a module whose 
bits report various combinations of the module; in a 
processor, register 8; in ae serial or parallel 
interface, the first register; in a DMA device, the 
fourth and seventh registers. } 


step -— the act of Causing a processor to execute a single 
instruction and then halt. | 
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start pointer - begin pointer. 


STRB - an INFIBUS signal which indicates that a master is 
transacting with a slave; used to strobe address and, 
on a write, data. ; 


strip - a set of instructions which are executed as a unit 
between references to the PID. 


SUE - System User Engineered; the name of the line of LEC 
parts which make up part of a Pluribus. 


SWB - SWitch Board; the front panel of the control panel. 


subnetwork - the collection of node computers (IMPs) and 
communication lines of a network which perform the 
actual routing and transmission of the data. 


synchronous - occurring at fixed time intervals. 


synchronous line -— a communications line where the timing 
information is derived from the transitions between 
data bits; data is usually sent in blocks. 


synchronizer — a device which resolves two asynchronous time 
references. 


system address space - the address space of common busses; 
seen directly by I/0 devices and indirectly by 
processors after mapping; 512K words long. 


system release notice - a document associated with each 
Pluribus system describing the location and 
configuration of each component. 


TAG - Timing And Gating; a LEC card which forms part of the 
memory. 


three phase wye —- the type of AC power that the Pluribus PDU 
requires; there are five wires: one is for protective 
(green) ground; one is common for the other three, each 
of which has a 117 volt AC potential with the common, 
but the phase of the legs is staggered by 120 degrees. 


throughput - the rate at which information can be processed. 


TI - usually refers to Texas Instruments Silent 700 
terminal. 


timer - a device, hardware or software, which watches over 
the activity of a part of the system; the timer is 
periodically reset by the occurrence of an event which 
Signifies correct operation and which should occur 
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periodically; should a specified time interval elapse 
without a reset, the timer will “time out" initiating 
some Souce ar action. | 


LLP = Terminal Interface message Processor; an IMP with 
built-in simple Host capabilities which allows users at 
terminals access to the network, obviating an external 
Host computer. | | 


TOD - Time Of Day; a modification to a pair of PPB boards to 
interface a Systron~-Donner clock. 


trap - either a QUIT or an ILLOP. 

very distant Host - a Host connected to the IMP via a 
communication link, with associated error detection and 
retransmission protocols. 

VDH - Very Distant Host. 

VISTAR - refers to an Infoton VISTAR terminal. 

watchdog timer -— cf. timer. 

window - one of the four 4K regions of processor address 
space which can be mapped onto a page of system address 
space. 

wire-wrap - a technology for making boards wherein 
connections are made by wrapping a wire around a pin 


POGeeS Jeg eCee to the component. 


oe the heave element of data; has 16 bits arid two parity 
bits; there are two bytes in a word. 


woven cable - a multiconductor cable constructed by weaving 
together several twisted pairs with a nylon thread. 


write — a bus transaction where the Gare flow is from master 
to slave. . 


20 


Report No. 2930 Bolt Beranek and Newman Inc. 


PLURIBUS DOCUMENT 2: SYSTEM HANDBOOK 


PART 4: INDEX 


Report No. 2930 Bolt Beranek and Newman Inc. 


PLURIBUS DOCUMENT 2: SYSTEM HANDBOOK 


PART 5: REPRINTS OF PAPERS 


ot 


Reprinted from — 


AFIPS — Conference Proceedings 
Volume 42 


© AFIPS PRESS 
Montvale, N. J. 07645 


A new minicomputer/multiprocessor 


for the ARPA network* 


by F. E. HEART, S. M. ORNSTEIN, W. R. CROWTHER, and W. B. BARKER 


Bolt Beranek and Newman Inc. 
Cambridge, Massachusetts 


INTRODUCTION 


Since the early years of the digital computer era, there 
has been a continuing attempt to gain processing power 
by organizing hardware processors so as to achieve some 
form of parallel operation.'? One important thread has 
been the use of an array of processors to allow a single 
control stream to operate simultaneously on a multiplic- 
ity of data streams; the most ambitious effort in this 


direction has been the ILLIAC IV project.’ **Another | 


important thread has been the partitioning of problems so 
that several control streams can operate in parallel. Often 
functions have been unloaded from a central processor 
onto various specialized processors; examples include 
data channels, display processors, front-end communica- 
tion processors, on-line data preprocessors—in fact, I/O 
processors of all sorts. Similarly, dual processor systems 
have been used to provide load sharing and increased 
reliability. Still another thread has been the construction 
of pipeline systems in which sub-pieces of a single 
(generally large) processor work in parallel on successive 
phases of a problem.’ In some of these pipeline 
approaches the parallelism is “hidden” and the user con- 
siders only a single control stream. 

In recent years, as minicomputers have proliferated, 
groups of identical small machines have been connected 
together and jobs partitioned quite grossly among them. 
Most recently, our group and several others have been 
investigating this avenue further, attempting to reduce 
the specialization of the processors in order to employ 
independent processors with independent control streams 
in a cooperative and “equal” fashion.®"# 

This paper describes a new minicomputer/ multipro- 
cessor architecture for which a fourteen-processor proto- 
type is now (February 1973) being constructed. The 
hardware design and the software organization include 
many novel features, and the system may offer significant 
advantages in modularity and cost/performance. The 


*This work was sponsored by the Advanced Research Projects Agency 
under Contracts DAHC15-69-C-0179 and FO8606-73-6-0027. 
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system contains an expandable number of identical proc- 
essors, each with some “‘private’’ memory; an expandable 
amount of ‘“‘shared”’ memory to which all processors have 
equal access; and an expandable amount of I/O interface 
equipment, controllable by any processor. The system 
achieves unusual modularity and reliability by making 
all processors equivalent, so that any processor may per- 
form any system task; thus systems can be easily config- 
ured to meet the throughput requirements of a particular 
job. The scheme for interconnecting processors, memo- 
ries, and I/O is also modular, permitting interconnection 
cost to vary smoothly with system size. There is no ‘‘exec- 
utive’ and each processor determines its own task alloca- 
tion. 

A key issue throughout most of the attempts at parallel 
organization has been the difficulty of partitioning prob- 
lems in such a way that the resulting computer pro- 
gram(s) can really take advantage of the parallel organi- 
zation. This issue is raised in its most serious form when 
the parallel machine is expected to work well on a great 
diversity of problems as, for example, in a time-sharing 
system. Our machine design has been developed under 
the highly favorable circumstances that (1) the initial 
application, and a prior software implementation in a 
standard machine, was well understood; (2) the initial 
application lent itself to fragmentation into parallel struc- 
tures; and (3) the design would be deemed successful if it 
handled only that one application in a meritorious fash- 
ion. However, we now believe that the design is advanta- 
geous for many other important applications as well and 
that it may herald a broadly useful new way to achieve 
increased performance and reliability. 

The machine has been designed to serve initially as a 
modular switching node for the ARPA Network? and, in 
the following section, we briefly describe the ARPA 
Network application and the requirements that the net- 
work imposed upon the machine design. In subsequent 
sections we discuss our choice of minicomputer, describe 
our system design in some detail, discuss certain of the 
more interesting characteristics of multiprocessor behav- 
ior, and summarize our present status and plans for the 
near future. 
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ARPA NETWORK REQUIREMENTS 


The ARPA Network, a nationwide interconnection of 
computers and high bandwidth (50 Kb) communication 
circuits, has grown during the past four years to include 


over 35 sites, with more than one computer at many sites.” 


The computers at each site, called Hosts, obtain access to 
the net via a small communications processor known as 
an Interface Message Processor or IMP.’° In order to 
permit groups without their own computer facility to 
access this powerful set of computer resources, a version 
of the IMP called a Terminal IMP allows, in addition, 
attachment of up to 63 local or remote terminals of a wide 
range of types.” 

As a considerable simplification, the job to be handled 
by an IMP is that of a communications processor. Arriv- 
ing messages must pass through an error control algo- 
rithm, be inspected to some degree (e.g., for destination), 
and generally be directed out onto some other line. Some 
incoming messages (e.g., routing control messages) must 
be constructed or digested directly by the IMP. The IMP 
must also concern itself with flow control, message assem- 


bly and sequencing, performance and flow monitoring, 


Host status, line and interface testing, and. many other 


housekeeping functions. To perform these functions an 
IMP requires memory both for program and for message _ 
buffers, processing power for executing the program, and | 


I/O units of various sorts for connecting to a variety of 
lines and devices. The original IMP, built around a 
Honeywell 516 processor with a 1 ws cycle time, could 
handle approximately three-quarters of a megabit per 
second of full duplex communications traffic. A later, 


smaller and cheaper (Honeywell 316) version handles 


about two-thirds as much traffic. 


As the network has grown and as usage has increased, a 


number of demands for improvement have led to the need 
for a new “line” of IMP machines. Our intent is to pro- 


vide a modular arrangement of flexible hardware from. 


which it will be possible to construct both smaller and less 
expensive IMPs as well as far more powerful IMPs. An 
important specific objective is to obtain an IMP whose 


communications bandwidth could be at least an order of © 


magnitude greater than the 516 IMP; such a high speed 
IMP would permit the direct connection of satellite cir- 
cuits or land T-carrier circuits operating at approxi- 
mately 1.3 megabits/second. 


It is also desirable to improve the present IMP design 


in a number of other areas, as follows. 


e Expandability of 1/0: The present IMPs permit 
connection to a total of only seven high-speed circuits 
and/or Host computers. We would like to permit a 
much greater fanout so that an IMP might be con- 

‘ nected to as many as 20 or more Host computers or 
to hundreds of terminals. This means that the num- 
ber of interface units should be expandable « over a 
wide range. 

e Modularity: A number of groups have wished to 
make a network connection from a single Host at a 


considerable distance (miles) from the nearest IMP. 
We feel that such Hosts should be locally connected 
to a very small IMP in order to preserve consistency 

_ and standardization throughout the network. There- 
fore, a goal of this new hardware effort is the provi- 
sion of a.small and inexpensive but compatible IMP 
which could serve to connect a single, distant | spur 
Host. 

@ Expandability of Memory: The new line of equip- 
ment is required for use in connection with satellite 
links (or longer faster links in general) and must 
therefore be able to expand its memory easily to 
provide the much greater buffer storage require- 
ments of such links. 

® Reliability: The new line of processors should be 
more reliable than the existing IMPs and ought to 
permit better self-diagnosis and simple isolation and 
replacement of failing units. 


Of the requirements posed by the ARPA Network 
application, the most central was to obtain an order-of- 
magnitude traffic bandwidth improvement. We first con- 


_ sidered meeting this requirement with highly specialized 


hardware, but the need to allow evolution of the commu- 
nications algorithms, as well as the ‘“‘bookkeeping”’ nature 
of much of the IMP task, militate against hardwired 
approaches and require the flexibility of a stored program 


computer. Thus we need a machine with an effective 


cycle time of 100 nanoseconds, a factor of ten faster than 
the present 1 us IMP. Realizing that a single very fast 
and powerful machine would be difficult to build and 
would not give us compatible machines with a wide spec- 
trum of performance, we began to consider the possibility 
of a minicomputer/ multiprocessor in order to achieve the 
flexibility, reliability, and effective bandwidth required. 
With the idea of a multiprocessor in mind we consid- 
ered the IMP algorithm to determine which parts were 
inherently serial in nature and which could proceed in 
parallel. It seemed difficult to process a single message in 


a parallel fashion: the job was already relatively short 


and intimately coupled to I/O interfaces. However, there 
was much less serial coupling between the processing of 
separate messages from the same phone line and no cou- 
pling at all between messages from different phone lines. 
We thus envisage many processors, each at work on a 
separate message, with the number of processors carefully 
matched to the number of messages we expect to encoun- 
ter in the time it takes one processor to deal with one 
message. With this simple image there seems to be no 
inherent limit to the parallelism we can achieve—the 
ultimate limit would be set by the size of the multiproces- 
sor we can build. 


CHOICE OF THE PROCESSOR 


In designing a multiprocessor for the IMP application, 
we found ourselves iteratively exploring two related but 
distinct issues. First, assuming that the problem of inter- 


connection could be solved, what minicomputer would be 
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a sensible choice from the price/ performance and physi- 
cal points of view? Second, and much harder: for any 
specific machine, how did the CPU talk to memory, how 
would multiple CPUs, memories, and I/O be intercon- 
nected to form a system, and how would the program be 
organized? 

Since the program for the existing IMPs was well 
understood, it was possible to identify key sections of that 
program which consumed the majority of the processing 
bandwidth. Then, for each sensible minicomputer choice, 
we could ask how many CPUs of this type would be 
needed to provide an effective 100 nanosecond cycle time; 
and given a price list, physical data, and a modest 
amount of design effort, we could define the physical 
structure and the price of the resulting multiprocessor. 
With this general approach, we examined the internal 
design of about a dozen machines, and actually wrote the 
key code in many cases. Using the fastest available mini- 
computers it was possible to arrive at configurations with 
only three or four processors; using the slowest choices, 
systems with 20 CPUs or more were required. 

If we defer the interconnection and contention prob- 
lems for a moment, it is interesting to note that “slow and 
cheap” may win over “‘fast and expensive”’ in this kind of 
multiprocessor competition to achieve a stated processing 
bandwidth. This is an especially happy situation if, as in 
our case, a spectrum of configurations is needed, includ- 
ing a very tiny cheap version. 

In considering which minicomputer might be most eas- 
ily adaptable to a multiprocessor structure, the internal 
communication between the processor and its memory 
was of primary concern. Several years ago machines were 
introduced which combined memory and I/O busses into 
a single bus. As part of this step, registers within the 
devices (pointers, status and control registers, and the 
like) were made to look like memory cells so that they 
and the memory could be referenced in a homogeneous 
manner. This structure forms a very clean and attractive 
architecture in which any unit can bid to become master 
of the bus in order to communicate with any other desired 
unit. One of the important features of this structure is 
that it made memory accessing ‘‘public’’; the interface to 
the memory had to become asynchronous, cleanly isolable 
electrically and mechanically, and well documented and 
stable. A characteristic of this architecture is that all ref- 
erences between units are time multiplexed onto a single 
bus. Conflicts for bus usage therefore establish an ulti- 
mate upper bound on overall performance, and attempts 
to speed up the bus eventually run into serious problems 
in arbitration.” 

In 1972 a new processor—the Lockheed SUE“%—was 
introduced which follows the single bus philosophy but 
carries it an important step further by removing the bus 
arbitration logic to a module separate from the processor. 
This step permits one to consider configurations embody- 
ing multiple processors and multiple memories as well as 
I/O on a single bus. The SUE CPU is a compact, rela- 
tively inexpensive (approximately $600 in quantity), 
quite slow processor with a microcoded inner structure. 


This slowness can be compensated for by simply doubling 
or trebling the number of processors on the bus; perform- 
ance is limited largely by the speed of the bus. With this 
bus architecture it becomes attractive to visualize multi- 
bus systems with a “bus coupling’? mechanism to allow 
devices on one bus to access devices on other busses. 

Similar approaches can be implemented with varying 
degrees of difficulty in systems with other bus structures, 
and we examined several approaches in some detail for 
those processors whose cost/performance was attractive. 
Rather fortuitously, the minicomputer which exhibited 
the most attractive bus architecture also was extremely 
attractive in terms of cost/performance and physical 
characteristics. This machine, the Lockheed SUE, would 
require fourteen processors to achieve the effective 100 
nanosecond cycle time, and we embarked on the detailed 
design of our multiprocessor on that basis. 


SYSTEM DESIGN 


Although our design permits systems of widely varying 
size and performance, in the interest of clarity we will 
describe that design in terms of the particular prototype 
now under construction. Our overall design is represented 
in Figure 1. We require fourteen SUE processors to obtain 
the necessary processing bandwidth, and we estimate that 
32K words of memory will be required for a complete 
copy of the operational program and the necessary 
communication buffer storage. The I/O arrangements 
must allow easy connection of all the communications 
interfaces, appropriate to the IMP job (modem inter- 
faces, Host interfaces, terminal interfaces) as well as 
standard peripherals and any special devices appropri- 
ate to the multiprocessor nature of the system. 

Some of the basic SUE characteristics are listed in 
Table I. From a physical point of view, the SUE chassis 
represents the basic construction unit; it incorporates a 
printed circuit back plane which forms the bus into which 
24 cards may be plugged. From a logical point of view this 
bus simply provides a common connection between all 
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TABLE I—SUE Characteristics 


16-bit word 
_8 General Registers 
A3.7 ps add or load time 
Microcoded . 
Two words/instruction typical 
8-146". 19” 18” chassis 
64K bytes addressable by a single instruction 
~$3K for: 1 CPU+4K Memory+Power, Rack, ete. 
200 ns minimum bus cycle time 
850 ns memory cycle time 
425 ns memory access time 


units plugged into the chassis. We are using these chassis 
for the entire system: processor, memory, and I/O. All 
specially designed cards as well as all Lockheed-provided 
modules plug into these bus chassis. With this hardware, 
the terms ‘‘bus’’ and “chassis”? are used somewhat inter- 
changeably, but we will commonly call this standard 
building unit a “bus.’” Each bus requires one card which 
performs arbitration. A bus can be logically extended (via 
a bus extender unit) to a second bus if additional card 


space is required; in such a case, a single bus arbiter | 


controls access to the entire extended bus. 

We can build a small multiprocessor just by plugging 
several processors and memories (and I/O) into a single 
bus. For larger systems we quickly exceed the bandwidth 
capability of a single bus and we are forced to multi-bus 


architecture. Then, from a construction viewpoint, our 


multiprocessor design involves assigning processors, 
memories and I/O units to busses in a sensible manner 


and designing a switching arrangement to permit inter- | 


connection of all the busses. Of course, the superficial 
simplicity of this construction viewpoint completely hides 
the many difficult problems of multiprocessor system 
design; we will try to deal with some of those issues in the 
following sections. 


Resources 


A central notion in a parallel system is the idea of a 
“resource,” which we define to mean a part of the system 
needed by more than one of the parallel users and there- 
fore a possible source of contention. The three basic 
hardware resources are the memories, the I/O, and the 
processors. It is useful to consider the memories, further- 
more, as a collection of resources of quite different char- 
acter: a program, queues and variables of a global nature, 
local variables, and large areas of buffer storage. 

The basic idea of a multiprocessor is to provide multi- 
ple copies of the vital resources in the hope that the algo- 
rithm can run faster by using them in parallel. The 
number of copies of the resource which are required to 
allow concurrent operation is determined by the speed of 
the resource and the frequency with which it is used. An 
additional advantage of multiple copies is reliability: if a 
system contains a few spare copies of all resources, it can 
continue to operate when one copy breaks. 


It may seem peculiar to think of a processor as a 
resource, but in fact in our system the parallel parts of 
the algorithm compete with each other for a processor on 
which to run. We take the view that all processors shall be 
identical and equal, and we go to some trouble to insure 
that this is in fact so. As a consequence no single proces- 
sor is of vital importance, and we can change the number 
of processors at will. A later section will describe how the 
processors coordinate to get the job done without a master 
of some sort. 


Processor busses 


A SUE bus can physically and logically support up to 
four processors. As more processors are added to a bus, 
the contention for the bus increases, and the performance 
increment per processor drops; but the effective cost per 
processor also drops, since the cost for the chassis, power 
supply, bus arbitration, etc., is amortized over the num- 
ber of processors. 

Roughly speaking, using two processors per bus loses 
almost nothing in processor performance, using three 
processors per bus loses significant efficiency, and adding 
a fourth processor gains less than half an “effective proc- 
essor.”’ After careful examination of the logical, economic 
and physical aspects of this choice, we decided to use two 
processors per processor bus, and we thus require seven 
processor busses in our initial multiprocessor system. 

The next question was how the processors should access 
the program. In our application, some parts of the pro- 
gram are run very frequently and other parts are run far 
less frequently. This fact allows a significant advantage to 
be gained by the use of private memory. When a proces- 
sor makes access to shared memory via the switching 
arrangement, that access will incur delays due to conten- 
tion and delays introduced by the intervening switch. We 
therefore decided to use a 4K local memory with each 
processor on its bus to allow faster local access to the 
frequently run code; these local memories all typically 
contain the same code. With this configuration and in our 
application, the ratio of accesses to local versus shared 
memory is better than three to one. This not only reduces 
contention delays for access to the shared memory but 
also cuts the number of accesses which suffer the delays. 

The final configuration of a processor bus is shown in 
Figure 2(a). The units marked ‘‘Bus Coupler’ have to do 
with our multiprocessor switching arrangement, which 
will be discussed below. 


Shared memory busses* 


The shared memory of our multiprocessor is intended 
to contain a copy of the program as well as considerable 
storage space for message buffering, global variables, etc. — 
Application-dependent considerations led us to select a 


* The terms “I/O bus’ and ‘memory bus” as used here and henceforth. 
are not the same as conventional I/O and memory busses. 
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Figure 2— Bus structures 


32K memory, but it is possible to contigure this memory 
on a single bus or to divide the memory onto several bus- 
ses. We first concluded that four logical memory units 
would be appropriate in order to reduce processor conten- 
tion to an acceptable level. Then, since the bus is consid- 
erably faster than the memories, it is feasible to place two 
logical memory elements on a single bus with almost no 
interference. Thus, we are planning two memory busses 
in the initial multiprocessor; the configuration of a 
common memory bus is shown in Figure 2(b). 


I / O b us Ss es 


The I/O system of the multiprocessor employs stand- 
ard SUE busses with standard bus arbitration units on 
those busses. Into the bus will be plugged cards for each 
of the various types of I/O interfaces that are required, 
including interfaces for modems, terminals, Host comput- 
ers, etc., as well as interfaces for standard peripherals. 
Our initial system has a single I/O bus and Figure 2(c) 
shows its configuration; the specialized units shown (a 
‘“‘Clock”’ and ‘‘Pseudo Interrupt Device’’) are system-wide 
resources that are used to control the operation of the 
multiprocessor. The I/O bus will also be the access route 
for the multiprocessor console; we plan to use a standard 
alphanumeric display terminal which can be driven by 
code in any processor, and no canventional consoles will 
be used. 


Interconnection system 


Our prototype multiprocessor is now seen to contain 


- seven processor busses, two shared memory busses and an 


I/O bus. To adhere to our requirement that all processors 
must be equal and able to perform any system task, these 
busses must be connected so that all processors can access 
all shared memory, so that I/O can be fed to and from 
shared memory, and so that any of the processors may 
control the operation and sense the status of any I/O unit. 

A distributed inter-communication scheme was chosen 
in the interest of expandability, reliability, and design 
simplicity. The atom of this scheme is called a Bus Cou- 
pler, and consists of two cards and an interconnecting 
cable. In making connections between processors and 
shared memory, one card plugs into a shared memory 
bus, where it will request cycles of the memory; the other 
card plugs into a processor’s bus, where it looks like 
memory. When the processor requests a cycle within the 
address range which the Bus Coupler recognizes, a 
request is sent down the cable to the memory end, which 
then starts contending for the shared memory bus. When 
selected, it requests the desired cycle of the shared 
memory. The memory returns the desired information to 
the Bus Coupler, which then provides it to the requesting 
processor, which, except for an additional delay, does not 
know that the memory was not on its own bus. Note that 
the memory access arbitration inherent in any memory 
switching arrangement is handled by the SUE Bus Arbi- 
ter controlling the shared memory bus, while the Bus 
Coupler itself is conceptually straightforward. 

One additional feature of the Bus Coupler is that it 
does address mapping. Since a processor can address only 
64K bytes (16 bit address), and since we wished to permit 
multiprocessor configurations with up to 1024K bytes (20 
bit address) of shared memory, a mechanism for address 
expansion is required. The Bus Coupler provides four 
independent 8K byte windows into shared memory. The 
processor can load registers in the Bus Coupler which 
provide the high-order bits of the shared memory address 
for each of the four windows. | 

Given a Bus Coupler connecting each processor bus to 
each shared-memory bus, all processors can access all 
shared memory. I/O devices which do direct memory 
transfers must also access these shared memories. These 
I/O devices are plugged into as many I/O busses as are 
required to handle the bandwidth involved, and bus cou- 
plers then connect each I/O bus to each memory bus. 
Similarly, I/O devices also need to respond to processor 


requests for action or information; in this regard, the I/O _ 


devices act like memories and Bus Couplers are again 
used to connect each processor bus to each I/O bus. The 
path between processor busses and I/O busses is also 
used in a more sophisticated fashion to allow processors 
to examine and control other processors; this subject is 
described in a later section. 

The resulting system is shown in Figure 3. One is struck 
by the number of bus couplers: P*I+I*M+P*M bus 
couplers are required for a system with P processor bus- 
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Figure 3—Prototype system 


ses, I I/O busses, and M memory busses. In the case of 
our initial multiprocessor, 23 are needed. | 
This modular interconnection approach clearly permits 
great flexibility in the number and configuration of bus- 
ses, and allows interconnection cost to vary smoothly with 
system size. We believe that this modular interconnection 
scheme also permits a complex hierarchical arrangement 


of busses. Actually the system exhibits a pronounced 
hierarchical structure already. A processor accesses the 
local memory when it needs instructions or local varia- 


bles. Two such processor-memory combinations form a 


dual processor, which can be regarded as a unit and 
which needs access to shared resources, such as global 
variables, free buffers, and I/O interfaces. When one 
copy of a resource can only support a limited number of 
users, it seems sensible to provide only the corresponding 
limited number of connections. If a multiprocessor of this 
type were to grow larger, the physical number of bus 
couplers as well as increasing contention problems might 
not permit the connection of each processor to all of 
common memory, but might instead require a multi-level 
structure where groups of processors were connected to an 
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intermediate level bus which was in turn connected to a 
centralized common memory. We have not explored this 
domain but feel it is an interesting area for future work. 


MULTIPROCESSOR BEHAVIOR 


Until the processors interact, a multiprocessor is a 
number of independent single processor systems: it is the 
interaction which poses the conceptual as well as the 
practical problems. If the various processors spend their 
time waiting for each other, the system degrades to a sin- 
gle processor equivalent; if they can usefully run concur- 
rently, the processing power is multiplied by the number 
of processors. If the failure of a single processor takes the 
system down, the system reliability is only the probability 
of all processors being up; if working processors can diag- 
nose and heal or amputate faulty processors and proceed 
with the job, the system reliability approaches the proba- 
bility of any processor being up. We now consider how to 
keep processors running concurrently, and then how to 
keep the system running in the case of module failure. 

The first problem in making the machines run inde- 
pendently is the allocation of runnable tasks to proces- 
sors, so that the full requisite power can be quickly 
brought to bear on high priority tasks. Our scheme for 
doing this rests on four key ideas: (1) We break the job up 
into a set of tiny tasks. (2) Our processors are all identi- 
cal, asynchronous, and capable of doing any task. (3) We 
keep a queue of pending tasks, ordered by priority, from 


which each processor at its convenience gets its next task. | 


(4) For speed and efficiency, we use a hardware device to 
help manage the queue. 

By breaking the job up into smaller and smaller tasks 
until each one runs in under 300 us, we effectively deter- 
mine the responsiveness of our system. Once started, a 
task must run to completion, but there will be a reconsi- 
deration of priorities at the beginning of each new task. 
We have chosen 300 microseconds as the maximum task 
execution time because this compromise between effi- 
ciency and resporisiveness is well matched to the execu- 
tion time of key IMP functions. 

By making the processors identical, we can use the 
same program in systems of widely varying size and 
throughput capability. Any processor can be added to or 
removed from a running system with only a slight change 
in throughput. The power of all processors quickly shifts 
to that part of the algorithm where it is most needed. 

By queuing pending tasks, we keep track of what must 
be done while focusing on the most important tasks. By 
using a passive queue in which the processors check for a 
new task when they are ready, we avoid some nasty tim- 
ing problems. Tasks may be entered into the queue at any 
time, either by a processor or by the hardware I/O 
devices. This approach is an extremely important depar- 
ture which avoids the use of conventional interrupts and 
the associated costs of saving and restoring machine state. 
Further, this approach neatly sidesteps the problem of 
routing interrupts to the proper processor. 


We could not afford a software queue both because it 
was slow to use and because processors would have been 
waiting for each other to get access to the queue. Instead 
we use a special hardware device called a Pseudo Inter- 
rupt Device (PID), which keeps in hardware a list of 
what to do next. A number can be written to the PID at 
any time and and it will be remembered. When read, the 
PID returns (and deletes) the highest number it has 
stored. By coding the numbers to represent tasks, and 
keeping the parameters of the tasks in memory, a proces- 
sor can access the PID at the end of each task and deter- 
mine very rapidly what it should do next. 


Contention 


Clearly, the PID must give any task to exactly one 
processor. This is guaranteed because the PID is on a bus 
that can be accessed by only one processor at a time and 
because the PID completes each transaction in a single 
access. This is an example of the more general problem 
that whenever two users want access to a single resource 
there must be an interlock to let them take turns. This is 
true at many levels, from contention for a bus to proces- 
sor contention for shared software resources such as a free 
list. When all the appropriate interlocks have been pro- 
vided, the performance of the multiprocessor will depend 
rather critically on the time wasted waiting at these inter- 
locks for a resource to become free. As discussed above, 
whenever conflicts become a serious problem one pro- 
vides another copy of the resource. We studied our system 
behavior carefully, noting areas of conflict, in order to 
know how many additional copies of heavily accessed 
resources to provide. Table II provides examples of 
delays due to various conflicts. Practically speaking, the 
curve of delay vs. number of resources has a rather sharp 
knee, so that it is meaningful to make such statements as 
‘“‘a memory bus supports eight processors”’ or “‘a free list 
supports eight processors.’ Of course, these statements 
are application related and depend on the frequency and 
duration of accesses required. 

With interlocks, deadlocks become possible (in both 
hardware and software). For example, a deadlock occurs 


TABLE II—Expected System Slowdown Due to Contention Delays 


Slowdown Cause 


5.5% Contention for a Processor Bus. 
3% Contention for the Shared Memory Busses. 
5% Contention for the Shared Memories. 


10% Contention for a single system-wide software resource, as- _ 


suming each processor wants the resource for 6 instruc- 
tions out of every 120 instructions executed. 


1.7% Contention for one of two copies of a system-wide software 


resource, as above. 

0.15% Contention for the parameters of a single 1.3 megabit 
phone line, assuming the parameters will be used for 160 
microseconds every 800 microseconds. 
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when each of two processors has claimed one of two 
resources needed by both. Each waits indefinitely for 
the othét’s resource to become available.'* Unless. there 
is a careful systematic approach to interlocks, deadlocks 
interlock, and require that a processor never compete for 
a resource when it already owns a higher numbered 
resource. It is not always practical or possible to do this, 
although we expect to be able to do so with the IMP algo- 
rithms. | 

An interesting example of a deadlock occurs in our bus 
coupling. To permit processors to access one another, for 
mutual turn on, turn off, testing, etc., the path connecting 
each processor bus with the I/O bus is made bi-direc- 
tional. Thus processors access one another via the I/O 
bus. In a bi-directional coupler, a deadlock arises when 
units obtain control of their busses at each end and then 
request access via the coupler to the bus on the other end. 
Because the backward path is infrequently used, we 
simply detect such deadlocks, abort the backward request 
and try again. 


Reliability 


We have taken a rather ambitious stand on reliability. 
We plan to detect a failing module automatically, ampu- 
tate it, and keep the system running without human 
intervention if at all possible. Critical to our approach is 
the fact that there are several processors each with pri- 
vate memory and thus each able to retreat to local opera- 
tion in the face of system problems. To reduce our vulner- 
ability further, power and cooling are provided on a 
modular basis so that loss of a single unit does not jeop- 
ardize system operation. We are only mildly concerned 
with the damage done at the time of a failure, because the 
IMP system includes many checks and recovery proce- 
dures throughout the network. 

The first sign of a failure may be a single bit wrong 
somewhere in shared memory, with all units apparently 
functioning properly. Alternatively, the failure may strike 
catastrophically, with shared memory in shambles and 
the processors running protectively in their local memo- 
ries. Against this spectrum we cannot hope for a system- 
atic defense; instead we have chosen a few defensive 
strategies. 

So long as a module is failing, recovery is meaningless. 
We must run diagnostics to identify the bad module, or 
see if cutting a module out at random helps things. We 
feel that identifying such a solid failure will be relatively 
easy. Since a processor without couplers is completely 
harmless, once we identifv a malfunctioning processor, we 
amputate it by oe off its bus couplers. We consid- 


runaway processor turning good 
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eee OlL This: is Saunlikely to begin with but we 
decided to make it even less likely by requiring a particu- 
lar 16-bit password to be used in turning off a coupler. A 
runaway processor storing throughout shared memory 
would need this password in its accumulator to acciden- 


tally amputate. Similarly we require a password for one 
processor to get at another’s local memory. 

Against intermittents we use a strategy of dynamic 
reinitialization. Every data structure is periodically 
checked; every waiting state is timed out; the code is 
periodically checksummed; memory transfers are hard- 
ware parity checked; memory is periodically tested; proc- 
essors are periodically given standard tests. Whenever 
anything is found wrong, the offending structure is initial- 
ized. Using this scheme we may not know what caused a 
failure, but its effects will not persist. In the most 
extreme cases we will need to reload all the program in 
main memory. Fortunately we have a communications 
network handy to load from. This technique of reloading 
has worked remarkably well in the current ARPA Net- 
work. Each processor has a copy of the reload program in 
its local memory, thus making loss of soa capability 
unlikely. 3 

We might seem to be vulnerable to memory or I/O fail- 
ures, particularly those involving the PID and the clock. 
If these modules fail it does indeed hurt us more, but only 
because we have fewer modules of these types in our sys- 
tem. If we provide redundant modules, the system can 
reconfigure itself to substitute a spare module for a failed 
one. Our design allows multiple I/O busses with multiple 
PIDs and clocks, and we could even have separate 
backup interfaces to vital communication lines on sepa- 
rate busses. : 

To summarize, the mainstay of our reliability scheme is 
a system continually aware of the state of things and 
quickly responding to unpleasant changes. The second 
line of defense consists of drastic actions like amputation 
and reloading. Assuming we can make all this work, we 
will have quite a reliable system, perhaps even one in 
which maintenance consists of periodic replacement of 
those parts which the system itself has rejected. 


STATUS AND NEAR FUTURE 


In February 1973, as this paper is submitted, we are 
very much in the middle of our multiprocessor develop- 
ment. Much progress has been made and we are increas- 
ingly confident of the design, but much work remains to 
be done. | 

The broad design is complete; all Lockheed-provided 
units (CPUs, memories, busses, etc.) have been delivered: 
prototype wire-wrapped versions of the crucial special 
modules have been completed, including the Bus Cou- 
plers, Pseudo Interrupt Device, clock, and modem inter- 
faces; and a multi-bus, multi-processor-per-bus assembly 
has been successfully tried with a test program. A sub- 
stantial program design effort has been in progress and 
coding of the first operational program has been started. 
We are still doing detailed design of some hardware, and 
we are still learning about detailed organizational issues 
as the software effort proceeds. An example of such an 
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Unless there is a careful systematic approach to interlocks, 
One technique is to assign a unique number to each resource for which 
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there is an interlock, and require that a processor never compete for a resource 
when it already owns a higher numbered resource. 


A New Minicomputer/ Multiprocessor for the ARPA Network 537 


area is: exactly how is it best for processors to watch each 
other for signs of failure? 

We currently anticipate the parts cost of the prototype 
fourteen-processor system, without communication inter- 
faces, to be under $100K. 

Hopefully, by the time this paper is presented in June 
1973, we will be able to report an operational prototype 
multiprocessor system. Beyond that, our schedule calls 
for the installation of a machine in the ARPA Network by 
about the end of 1973. We also plan to construct many 
variant systems out of this kit of building blocks, and to 
experiment with systems of varying sizes. As part of this 
work, we plan to concentrate on the very smallest version 
that may be sensible, in order to provide a minimum cost 
IMP for spur applications in the ARPA Network. 

As the design has proceeded, our attraction to the gen- 
eral approach has increased (perhaps a common malady), 
and we now believe that the approach is applicable to 
many other classes of problems. We expect to explore 
such other applications as time permits, with initial 
attention to two areas: (1) certain specialized multi-user 
systems, and (2) high bandwidth signal processing. 

With our presently planned building blocks, although 
we do not yet know what will limit system size, we do not 
now see any intrinsic problem in constructing systems 
with fifty or a hundred processors. As improvements in 
integrated circuit technology occur, and processors and 
memories become smaller and cheaper, organization and 
connection become the paramount questions in multipro- 
cessor design. We expect to see many attempts at multi- 
processors, and are hopeful that the ideas embodied in 
this design will help to steer that technology. Perhaps 
minicomputer/ multiprocessors will soon represent real 
competition for the various brontosaurus machines that 
now abound. 
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Abstract 


The BBN multiprocessor has gone from conception to prototype over 


the past year. 


It is highly modular at several logical and physical 
levels and will soon be a new IMP in the ARPA Network. 


It is very 


flexible both in the range of bandwidths it can handle and the 
number and type of interfaces it can accommodate. 


1. INTRODUCTION 


Last year we presented a paper which de- 
scribed the multi-processor we were then 
setting out to build as a new IMP for the 
ARPANET [1,2]. Much has been accomplished 
in this past year and we report here on 
progress made as well as on some important 
features of the system that have evolved. 
Familiarity with the earlier paper is 
assumed in what follows. 


The architecture, as previously described, 
is highly modular and allows for IMPs of 
greater or lesser processing power than the 
present 516/316-based IMPs, as well as for 
many more and more varied phone line and 
Host interfaces. The hardware consists of 
busses joined together by special bus cou- 
Olers of our design. There are processor 
busses each of ‘which contains two proces- 
sors, each in turn with its own "private" 
4K memory to store frequently run code. 

The more processor busses, the greater the 
system processing power. There are memory 
busses to house the segments of multiported 
"common" memory — the more memory busses, 
the more memory ports. Finally, there are 
I/O busses which house device and line con- 
trollers as well as a special (priority or- 
dered) task disburser which replaces the 
traditional priority interrupt system. The 
latter allows equality among the processors 
so that if some fail the rest can continue 
to run all system tasks, albeit at reduced 
capacity. 


2. DESIGN ISSUES 


In this section we describe features we 
have designed into the system, some of the 
more interesting of which relate to reli- 
ability issues. 


2.1 ADDRESSING & LOCKING 


The Lockheed SUE, with a 15-bit word ad- 
dress, can address up to 32K words. A 
1.5-megabit line running over a 1/2 sec. 
round trip satellite channel holds 750,000 
bits or about 50,000 words, copies of which 
must be held in the IMP for possible re- 
transmission. Address expansion is thus 
inescapable and to allow for several such 
lines and be reasonably unbound by address 
space, we have allowed for half a million 
words. The bus coupler serves as the vehi- 
cle for address expansion. 8K of a proces- 
sor's address space are used for direct 
references to its private memory. (Al- 
though we expect to use only 4K, 8K has 
been set aside to allow for growth.) An- 
other 8K is used principally for addressing 
system I/O (on the up to four I/O busses). 
We assign 8 addresses to each I/O device 
for pointers and status and control regis- 
ters; 960 devices can be accommodated in 

eC Oleer 


16K of each processor's address space is 


mapped through the couplers to common memo-_ 


ry. At the processor end of each coupler 
are four program-settable map registers for 
each possible processor on the bus. (We 


*This work was supported by the Advanced Research Projects Agency under Contracts 
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“at a particular site at a given time. 


expect to use only 
but up to four are 
registers expand a 


two processors per bus 
allowed for.) These map 
15-bit address to a 19- 
bit system address on the memory busses. 

By use of the maps, each processor can thus 
access, at any one time, four 4K pages in 
System address space. Read accesses 
through a particular one of these windows 
are turned by the coupler into read-clear 
operations, thereby providing the indivisi- 
ble test-and-modify operation required for 
program interlocking in a multi-processor. 
(The processor itself presently lacks such 
an instruction.) 


2.2 ACCESS ENABLING 


The coupler paths that connect processor 


busses into memory and I/0 busses have pro-_ 


gram settable enabling switches at their 
far (memory and I/O) ends, thus permitting 
processors to be.cut in and out of the sys- 
tem. To allow processors to access one 
another and to permit reloading as discuss- 
ed below, we have provided reverse paths in 
the processor to I/9 couplers which also 
have enabling switches. Normally the for- 
ward paths to memory and I/O are turned on 
and the backward paths are shut off. Since 
these paths represent a hazard whereby a 
"Sick" processor or device could damage 
healthy processors, we have arranged that 
only by storing a password at the proper 
address can a switch be changed. This 
greatly reduces the probability that a ber- 
serk processor painting memory will affect 
the pathy. A processor carn neither enable 
nor disable its own access paths but one 
processor, deciding that another is sick 
and. .should. be-eliminaved. from tne system, 
can amputate the bus of the offending pro- 
cessor. It can be similarly reinstated 
later. 


The logic upon which amputation decisions 
are based is not yet fully understood and 
will be worked out as experience grows. We 
expect to require all processors to execute 
periodic healthiness-proving tasks. A 
regular system task, performed by any free 
processor, verifies that all processors 
have passed their tests and amputates any 
unhealthy one(s). Protective embellish- 
ments easily suggest themselves and we ex- 
pect to do what seems necessary. 


25 DISCOVERY 


- The operational program implements the IMP 
algorithm with whatever hardware is working 
The 

' program discovers the hardware configura- 
tion as Tollows:.. Memory is Tound by trying 
to access it; a failure interrupt results 
if Memory is not there. Processors are 
found by accessing a register whose re- 


sponse indicates if the processor is absent, 


running or halted. I/0 Devices are found 
by reading the lst word of every possible 


device in I/O space —a failure interrupt 
means no Device, a response returns a 
unique 16-bit device type. Any parameters 
needed to run the devices are available as 
status words in the 8-word block. . It is 
somewhat harder to find where the bus 
boundaries are, but they too can be found 
by searching for the bus coupler disable 
switches. In the event that there is some 
property we cannot otherwise discover, we 
have set aside 3 registers (associated with 


the clock device) to hold this information. 


For example, the IMP number (used for net- 
work routing) is contained in 8 bits of 
these registers. 


The Discovery logic is not an initializa- 
tion phase; rather the program periodically 
runs through the Discovery logic and recon- 
figures whenever a change occurs. it thus 
automatically adapts the IMP algorithm not 
only to the wide variety of possible con- 
figurations but also to those which contain 
broken components. 


2e4. /PARITY 


At present the memories we are using do not 
store parity; however, we have built into 
our system design (and into the hardware) 
mechanisms to incorporate parity. These 
mechanisms have been tested with prototype 
parity memory and we have recently ordered 
parity memories for our production machines. 
We use a novel parity computation based not 
only upon the contents of a word but also 
on its address. The scheme also detects 
both "all ones" and "all zeros" failures. 
For writes to common memory, parity is com- 
puted at the processor end and fed, via the 
coupler, to the memory where it 1s stored 
with the word. Reads from memory fetch 
this stored parity, which is compared to a 
recomputed parity at the processor end of 
the coupler, thus checking both the memory 
and coupler paths in both directions. For 
units on the I/O bus, in order to check the 
coupler paths, a special card computes and 
transmits parity for all words being read 
from the I/O bus by the processors and 
checks parity on all words arriving .from 
processor busses. 


2.5 . RELOADING 


At present we use paper tape to load the 
system. The operator starts a processor 
which, from tape, loads its own private 

memory, its map registers and thereby any 
or all of common memory. It aiso loads, 

using backward coupling, the private memo- 
ries on all other processor busses in the 
system. After the memory has been loaded, 
a startup procedure is executed which fi- 
nally turns on the other processors. _ 


Since all crucial switches, parameters, 
registers and control flip flops have been 
made addressable by reads and writes, load- 


ing the system and starting it up can be 
done by externally force feeding it with 
the right set of addresses and data. Al- 
though we presently use paper tape in con- 
Junction with a bootstrap ROM executed by a 
processor for this purpose, we are planning 
to construct a means whereby the system can 
be force fed directly from the network. 

The mechanism for this is a device on the 
I/O bus which monitors phone lines from ad- 
jacent IMPs looking for a special format 
Which signals arrival of reload informa- 
tion. The card then performs the reload by 
executing store type bus cycles using the 
reload data. 


This sort of operation, which looks forward 
to elimination of paper tape, switches, and 
other operator dependent functions, is ap- 
propriate to the IMP job. If a running 
system fails, as viewed from the net, the © 
first step is to send it a regular "for IMP" 
message which causes a standard system re- 
start to be attempted. If that seems not 
to work, the next step is to send another 
regular message trying to activate the 
reload-from-the-net code in hopes that it 
Le: Still antact. -Only av that: Tails. would 
one attempt to force a full restart from 
scratch, in which case the special card de- 
scribed above is called into play. The 
first data sent halts the processors in or- 
der to stop any interfering activity. Then 

“the reload-from-the-net code is refreshed 

and finally a processor restarted running 
that code which then completes reload via 
the normal packet mechanism. 


2.6 MECHANICAL MODULARITY 


We have settled on a modular mechanical 
Structure well matched to the modular logi- 
cal structure of the system. This struc- 
ture 1s important in that it allows easy 
construction of systems of varied size and 
permits repair of parts of a system while 
the rest of it continues to operate. The 
basic unit is a cooling module which houses 
either 1) a 16-slot bus complete with its 
own power supply, 2) a 24-slot bus without 
power, or 3) a power supply for such a 24- 
Slot bus. These units, each with its own 
set of fans, sit. on rails in -a@-vertical 
tier in a rack, five of them filling a stan- 
dard height rack. (The 14-processor system 
requires three racks.) Figure 1 shows how 
the cooling modules stack. Air flow is from 
back to front so that racks placed beside 
one another do not directly heat each other. 
A tilted pan at the bottom of each module 
Separates the air flow between stacked mod- 
ules, thus eliminating chimney effects. 
Cards plug in from the front and all device 
and coupler cables also connect on that 

» Side. An entire unit can be removed to the 

rear for repair or replacement of the bus, 
rans, etc. — all without disturbing opera- 
tion of the remainder of the system. 
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3. THE TEST PROGRAM 


The primary design objective of the test 
program is to exercise all of the hardware 
as intensively and extensively as possible, 
detecting all failures and reporting them 
precisely and comprehensibly. Extensive 
testing implies a wide variety of test 
modules; intensive testing implies permit- 
ting the entire computational power of the 
system to be focused on individual compo- 
nents at times. These objectives led to 
the selection of a system based on process- 
es, analogous to a time-shared system's 
jobs. Processes are not tied to proces- 
sors; a given process will switch rapidly 
from one processor to another. Nor isa 
process in general tied to a specific copy 
of code; like time-shared jobs, processes 
share a single copy of sections of pure 
procedure, 


There are four types of processes: the 
"system" processes, including the clock, 
timeout, and type-out processes; the de- 
vice-specific processes, which are tied to 
particular I/O devices, two processes per 
device; the "GART" (Get A Random Test) 
processes, which select a test at random 
from a table of tests to be performed; and 
a dummy process, whose sole purpose is to 
assure that there is always a runnable 
process. 


Each GART test is designed to test a par- 
ticular element or feature of the system. 
These range from standard processor and 
memory tests (the latter are also useful 
for checking bus couplers) to exercising 
the various bus coupler switches and maps. . 
The I/O devices are kept busy by circula- 
ting various data through them. 


4. WHERE WE STAND 


Although the system uses Lockheed SUE pro- 
cessors, busses, memories, etc., we have so 
far designed and built nine BBN card types 
for the system: three coupler cards for 
each of the three bus types, a full-duplex 
memory channel card, a Host interface card 


(which operates at speeds up to 1.5 mega- 
bit), transmit and receive modem cards, the 
pseudo-interrupt card and a clock card. 
These designs are virtually all finalized 
and many are in production (printed circuit 
or similar) form. 


We are presently finishing the design of 
two other cards: the first of these is the 
parity checking card for the I/O bus de- 
scribed above under the discussion of par- 
ity. The second is a checksum/block-trans- 
fer card which flows a block of memory 
through itself computing a checksum as it 
goes. This is used to checksum critical 
code from time to time [3], to compute 
checksums for network end-to-end checking 
of messages, and other useful checking pur- 
poses. A transfer mode can be enabled so 
that it can also be used to move blocks of 
information about in memory (checksumming 
as it goes if desired). In addition we are 
presently embarking on modifications to the 
modem transmit and receive cards which will 
allow them to deal with 1.5 megabit lines 
and design of the special interface which 
monitors incoming inter-IMP lines watching 
for reload information as described above. 


At present we are running several systems. 
Two small systems are being used for test- 
ing and debugging of the IMP program. 

These are sometimes run as separate single 
bus IMP systems which are connected togeth- 
er with our prototype 516 IMP into a three- 
node network. At other times the two bus- 
ses are combined into a single system using 
a bus coupler. In this case one bus is 
used as a dual processor bus and the other 
as a combined memory and I/O bus. This 
system then works with the 516 IMP to form 
a e-node net. 


The growing prototype 14-processor system 
presently consists of three dual processor 
busses, two memory busses and one I/O bus. 
We have grown up to this system gradually 
but it now operates with sufficient reli- 
ability under stress (shaking of cables, 
margining power supplies, shuffling of 
cards, etc.) that we are presently in the 
process of building toward the full proto- 
type (i.e., adding the 2nd I/O bus and the 
remaining four processor busses). By mid- 
1974 we hope to have two production copies 


of this large prototype working in the net- 


work. During 1974 we plan also to design 


satellite modem interface cards and to pro- 


n +A adtonA 
 Quece and deliver three moderate sized sys- 


-tems with satellite capability ra]. 


Se The basic IMP system program is up and run- 
ning in multi-processor form, that is, with 


processors picking tasks up via the pseudo- 
interrupt system and using locks to prevent 
interfering accesses to resources. 
it has been run only.with a two-processor 
system, but it will shortly be put on the 
larger prototype. The inner parts of the 


So far 


system, store and forward, Host, task, etc., 
seem solid. The work that remains is in im- 
plementing the system maintenance, monitor- 
ing, and debugging functions (i.e., system. 
DDT, periodic status reports, etc.). This 
coding is about half done and needs finish- 
ing as well as debugging. The network er- 
ror recovery code is ready for debugging. 
The special reliability code which keeps 

the system up when parts of the hardware 
fail is being designed. 


Much work must be done in the present net- 
work to accommodate the advent of the new 
line of machines. For example, the whole 
reloading mechanism must be changed since 
one's neighbor may now be very different 
from one's self. The network must there- 
fore be able to pass core load images 
packet-by-packet to an immediate neighbor 
of the machine needing reloading. 


Our small IMP is built on a single logical 
bus (consisting of two separate physical 
busses connected by an extender) which com- 
bines memory, processor and I/O. This sys- 
tem embodies none of the special reliability 
stemming from multiple hardware copies but 
is the least expensive version available. 
Small reliable systems are another matter 
and require, in general, doubling the sys- 
tem to provide complete redundancy of parts 
to allow for any single failure. Such sys- 
tems may prove to be one of the more signif- 
icant outgrowths of this development effort. 
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INTRODUCTION 

As computer technology has evolved, system architects haa 
continually sought new ways to exploit the decreasing costs of 
SVSUCM -COMDOnenUS.. One approach: has Deen TO pull Popethner collec= 
GLOns: OF ULES ACO. MULL processor systems.~ Usually the objec- 
tives have been to gain increased operating power through paral- 
lelism and/or to gain increased system reliability through re- 
dundancy. 

In 1972, our group at Bolt Beranek and Newman started to de- 
sign a new machine for use as a switching node (IMP) in the ARPA 
Network.°?2 The machine was to be capable of high bandwidth, in 
order to handle the 1.5-megabaud data circuits which were then 
planned for the network. It was to have a high fanout to Host 
computers connected at a node. It was to come in all sizes (of 


processing power, memory, I/O) SO. that one coula conti eure an 
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individual IMP to meet the requirements of its particular location 
in the Senne, and change that configuration easily should the 
requirements change. Most of all, it was to be reliable. 

The family of machines we have produced which meets these 
goals has been named the Pluribus line. The machines are highly 
modular at several levels and have a minicomputer/multiprocessor 
architecture. Although the largest configuration we have put to- 
gether so far contains only 13 processors, we believe there are 
no inherent problems with eonetderanay larger systems. The struc- 
LUre:..eana details of some of the Hardware are described in. earlier 


445 


papers. Familiarity with these papers will be helpful in under- 
standing the present paper, which focuses on the issue of reli- 
ability. We believe that reliability will become an increasingly 
common concern as multiprocessors pacers fom commonplace, and we 
believe that we have gained some interesting insights ne > Gie 
SOlULLOM .Of “Gis problem. 
THE: MULTIPROCESSOR ARCHITECTURE 

A novel feature of our design is the consistent treatment of 
all processors as equal units, both in the hardware and in the 
software. There is no specialization of processors for particular 


system functions, and no assignment of priority among the proces- 


sors, such as designating one as master. We chose to distribute 


put also the multiprocessor control and reliability jobs, treating 


all jobs uniformly. We view the processors as a resource used to 
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advance our algorithm; the identity of the processor performing a 
particular task is -of no importance. Programs are written as for 
a Single processor except that the algorithm includes interlocks 
necessary to insure multiprocessor sequentiality when required. 
The sof'tware of our machine consists of a single conventional 
program run by all processors. Each processor has its own local 
copy of about one quarter of this program and the remaining three 
quarters is in commonly accessible memory. 

Hardware Structure 

Rotana toy was amain concern in planning the hardware 
architecture. Although we tried to build the individual preces 
solidly, our main goal was to provide hardware which could be 
exploited: by the Dropram: vo survive che fLatlure of any t1ndividual 
component. 

The hardware consists of busses joined together by special 
bus couplers which allow units on one bus to access those on 
another. Bach bus, together with 10s. own power supply and cool-= 
ing, is mounted in its own modular unit, permitting flexible 
variation in the size and structure of systems. There are 
processor busses each of which contains two processors, each in 
turn with its own local 4K memory which stores frequently run 
and recovery-related code. There are memory busses to house the 


segments of a large memory common tc all the processors. Finally, 
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there are I/O busses which house device controllers as well as 
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certain central resources such as system clocks and special 
(priority-ordered) task disbursers which replace the traditional | 
priority interrupt system. About half of the machine consists 

of standard parts from the Lockheed SUE line; the remainder is 

of special design. 

As emphasized in our initial Benen ae were fortunate To 
have a very specific job in mind as we designed the system. This 
enabled us to place specific bounds on the problems we sought to 
solve. For example, the proposed initial setting within a com- 
munications network means that outside entities (neighboring 
eommunicatvilons processors. ieebe. users, etc.) may help to notice 
that things are voing wrong. It also means that recovery assist- 
ance: 2S: potentially -avaitabie from the Network Control Center 
(NCC) through the Hemmer! The system is designed generally 
to avoid reliance upon external help, but upon occasion such help 
is useful and therefore we have provided methods for permitting 
the system to be forcibly reloaded and restarted via the network. 
pottware: Structure: . 

The problem of building a packet-switching store-and-forward 
communications processor (the IMP) lends itself especially well 
to parallel solution since packets of data can be treated in- 
dependently of one another. Other functions, such as routing 


computations, can also be performed in parallel. 
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The program is first divided into small pieces, called 


strtps, each of which handles a particular aspect of the job. 


When a task needs to be performed, the name (number) of the ap- 


DrOpraace SUri1D 1S put On & 
sor, when 10 2S not running 
When a strip number appears 
cessor will take 20. orf the 
strip. We try to break the 
a minimum of context saving 

The number assigned to 


Che task it performs. When 


queue of tasks to be run. Each proces- 
a strip, repeatedly checks this queue. 
On: BNE CUueUes,, TNE NeExXU-avaiiap ve pro 
queue and execute the corresponding 
program into strips in such a way that 
1s. necessary. 

each. SGPao rer iecus: The DrLoracy -OF 


a WrOCeSSor CNneGKs Toe task queue, act 


takes the Highest: PriOrLuy. Waltine jobs ..Sance all ‘processors 


access This Vueue TRrequenvsy. COnvenULom: For 16 18: arery hien: 


We therefore built a hardware device called the Pseudo Interrupt 


Device (PID) which serves as a task queue. A single instruction 


aLlows, CAC: DPehest. DRIOriyy 


task to be fetched and removed from 


the queue. Another instruction allows a new task to be put onto 


the queue. All contention 1s arbitrated by standard bus logic 


hardware. 


he Denetlh Of Strips 2s eoverned by now: lone priority. tasks 


can wait if all the processors are busy. The worst case arises 


when ai]. precessors Nave use Deeun. the Jonsest. strip. in: the 


IMP application, the most urgent tasks can afford to wait a maxi- 


mum of 400 microseconds. Therefore, strips must not generally be 
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longer than that. 

An inherent part of multiprocessor operation is the locking 
of critical resources to enforce sequentiality when neeaueey 
A load-and-clear operation provides our primitive locking facility. 
To avoid deadlocks, we priority-order our resources and arrange 
that the software not lock one resource when it has already locked 
another of lower or equal priority. 
Status 

During the early spring of 1974 a prototype 13-processor 
system was constructed. As this paper is being written (in the 
fall of 1974) two production copies have been constructed and are 
running. Each contains 13 processors, two memory busses, and two 
I/O busses. These machines have been connected intermittently 
into the ARPA Network for testing purposes and operational instal- 
lation in the neLOEk is anticipated shortly. A single processor 
has been running on the network for an extended period in order to 
validate Serr onianee during rouvine Operation... ‘Inree Savellive 
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IMP configurations” are presently under construction as well as a 
non-IMP configuration designed to provide highly reliable pre- 
processing and forwarding of seismic data to processing and storage 
centers. 

RELIABILITY GOALS 


Since the term "reliable system" can have many different meanings, 


it is important to establish clearly just what we are and what we are 
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not trying to achieve. We are not trying to build a non-failing 
device (as in ae instead, we are trying to build a system which 
will recuperate automatically within seconds, or at most minutes, 
following a failure. Furthermore, we want the system to survive 
not only transient failures but also solid failures of any single 
component. In many cases (such as the IMP job) it is not necessary 
GO -Operace: COnvinuously and pertrecvtly= 10, Ssultices to operate cor= 
rectly MOsG Or the time. so tone as outages. are Intrequcnt.. Kept 
brief, and fixed without human intervention. 

How one copes with infrequent brief outages depends on what 
One. AS “tryine TO do. For tasks not tightly coupled to real-time 
requirements (e.g., for many large numerical computations), a 
Simple Gevice 1s TO Choose eneckpoinis @tl- whiten: VO record. “Une 
Stace Of the System so that one -can always recover by restariing 


COM TING: ecko. jus rece lie. an outage. t+? 


The IMP sys- 
tem happens to be embedded in a larger system which is quite for- 
oivine. (This: nom an iuneommom Situatron.) Thus prier cutages 
of a few seconds are tolerated easily, and outages of many seconds, 
while causing the particular node to become temporarily unusable, 
will not in general jeopardize operation of the network as a whole. 
Occasionally, despite all efforts, the system will break so 
Cavasvropnically that ic: wild De unable to: recover, ‘Our coal 1s 


to reduce the probability of such total system failure to the 


probability of a multiple hardware failure. We do not try to 
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protect against all possible. errors; ee re procedures will 
fail to detect errors whose probability of. occurrence is suf- 
ficiently low. For example, all code is periodically checksummed 
using a 16-bit checksum. A failure that does not disturb the 
validity of the checksum may not be detected. We do not mind if - 
a failure renders large sections of the machine unusable or inac- 
cessible, providing enough remains to run the system. The presence 
of runnable hardware, however, is not sufficient to guarantee that 
operation will be resumed; in addition, the software must be able 
to survive the transients accompanying the failure and adapt to 
the remaining hardware. This may include combating and overcoming 
active failures (for example, when an element such as a processor 
goes berserk and repeatedly writes meaningless data into memory). 

Ail code is presumed to be debugged ~-- i.e., all frequently 
occurring problems will have been fixed. On the other hand, we 
must be able to survive infrequent bugs even when they randomly 
destroy code, data structures, etc. 

In order to avoid complete system failure, a failed component 
must be repaired or replaced before its backup also breaks. The 
system must therefore report all failures. The actual repair 
and/or replacement will of course be performed by humans, but this 
will generally take place dno Bete the system has noted the 
failure and recontigures 1tsell Go bypass The tailed. modules .the 


ratio of mean-time-to-repair to mean-time-between-failures will 
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determine overall system reliability. It must also be possible 
to remove and replace any component while the system continued to 
run. Finally, the system should absorb repaired or newly intro- 
duced parts gracefully. 
STRATEGIES 

in) order tO understand our system 20 2s convenient to: con- 
Sider the strategies used to achieve our goals in two parts which 
more or less parallel the traditional division into hardware and 
software. The first part provides hardware that will survive any 
Single failure, even a solid one, in such a way as to leave a 
potentially runnable machine intact (potentially in that it may 
need resetting, reloading, etc.). The second part provides all 
of the facilities necessary to survive any and all transients 
stemming from the failure and to adapt to running in the new hard- 
ware configuration. 
Appropriate Hardware 

We have two basic strategies in providing the hardware. The 
first is to inelude extra copies of every vital hardware resource. 
The second is to provide sufficient isolation between the copies 
so that any single component failure will impair only one copy. 

To increase effective bandwidth in multiprocessors, multiple 
CODES OL heavily utilized resources are normally provided. For 
reliability, however, ali resources critical to running the 


algorithm are duplicated. Where possible the system utilizes 
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these extra resources to increase the bandwidth of the system. 

It is not sufficient merely to provide duplicate copies of 
a particular resource; we must also be sure that the copies ane 
not dependent on any common resource. Thus, for example, in ad- 
dition to providing multiple memories, we also include logically 
independent, physically modular, multiple busses on which the 
memories are distributed. Each bus has its own power supply and 
cooling, and may be disconnected and removed from the racks for 
Serene while the rest of the machine continues to run. 

All central system resources, such as the real time clock 
and the PIC, are duplicated on at least two separate I/O busses. 
All connections between bus pairs are provided by separate bus 
couplers so that. a coupler exitiuce ath Gussie at most the two 
busses it iS connecting. 

Non-central resources, such as individual I/O interfaces, are 
generally less critical. Provision has been made, however, tO 
connect important lines to two identical interface units (on 
separate I/O busses) either of which may be selected for use by 
the program. 

To adapt 60 different hardware configurations, the software 
must be able to determine what hardware resources are available 
to it. We have made it convenient to search for and locate those 
_ resources which are present and determine the type and parameters. 


of those which are found. 
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To allow for active failures, all bus couplers have a program- 
COnvurollaple swaten. That. inhibits: transactions Via Ghav coupler. 
Thus, a bus may be effectively "amputated" by turning off all cou- 
plers from that bus. This mechanism is protected from capricious 
use by requiring a particular data word (a password) to be stored 
Ii a “COonuro. register of the bus coupler. Naturally an amputated 
processor is prevented from accessing these passwords. 

VWinaliy, alvnough a.common: reser. tine 16° normally considered 
essential, we have avoided such a line since a Siweie failliure on 
its driver could jeopardize the entire system. There is thus no 
central point (not even a single power switch) where one can gain 
CONTrOL Of Une GnUire ‘SsyStem at Once. Instead. we rely on. reser 
Ling @. SSCTEON “au a time UsSine passwords. 
software survival 

With the 2beove Teavures:. the Fiuribus Daraware Can exper ence 
any single component fallure- and still present a. runnable system. 
One must assume that as a consequence of a failure, the program 
may have been destroyed, the processors halted, and the hardware 
put in some hung state needing to be reset. We now investigate 
the means used to restore the algorithm to operation after a fail- 
ure. The various. Techniques for doing this’ may be classi Tied mages 
three broad strategies: keep it simple, worry about redundancy, 


and use watchdog timers: througnouv. 
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It is always good to keep a system simple, for then one 
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has a fighting chance to make it work. We describe here three 
system constraints imposed in the name of simplicity. 

First, as mentioned above, we insist that all processors 
be identical and equal: they are viewed only as resources used 
to advance the algorithm. Each should be able to do any system 
task; none should be singled out (except momentarily) for a parti- 
‘cular function. The important thing is the algorithm. With this 
view it is clear that it is simplest if the algorithm is accessible 
to all processors of the system. A consequence of this is that 
the full power of the machine can be brought to bear on the part 
of the algorithm which is busiest at a given time. 

One might argue that for some systems it is in fact simpler 
(or more sppiedenes to specialize processors to specific tasks. 

One could, in such a case, then duplicate each different type for 
reliability. With that approach, however, one must worry about 
the recovery of several different types of units, and all the pos- 
sible interactions between them. We consider the recovery problem 
for a group of identical machines formidable enough. 

One consequence of treating all processors equally is that 
a program can be debugged on a single machine up to the point 
where the multiple machine interaction matters. Once this has 
been done, we have found that processor interaction does not 
present a severe additional debugging problem. On the other hand 


finding routine software bugs when a dozen machines are running 
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iS ey WET ends prop Lem. 

A second characteristic of our system which arose from a 
desire to keep things simple is passivity. We use the terms 
active and passive to describe communication between subsystems 
In Which the receiver is expected to put aside what. it is doing 
and respond. The quicker the required response, the more active 
Che interaction. In general, the more passive the communication, 
the simpler the receiver can be, because it can wait until a con- 
venient time to process the communication. On the other hand 
the slower response may sone tienes: Shines for the sender. We 
believe that there isa net: gain in using more passive systems. 
An example of this 1s our decision to make the task disbursing 
mechanics (the PID) a passive device. Neither the hardware inter- 
faces: Hor OT her Drocessors. Cell. -a@ processor what Vo. ao; ravher. 
processors ask the PID what should be done next. There are Sone 
costs to such a passive system. The resulting slower responsive- 
ness has necessitated additional buffering in some of our inter- 
faces. tn addition, the program must. recularly break. from tasks 
being executed to check the PID for more important tasks. 

The alternatives, however, are far worse. In amore active 


system, for example one which uses classical priority interrupts, 


it is difficult to decide which processor to switch to the new task. 


Furvhnermore,. 10-28-almost ampossible to preserve the COnvext of 4 
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processor while making such a switch because of the interaction 
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with the resource interlocking system: The possibilities: for 
deadlocks are frightening, and the general mechanism to resolve 
them cumbersome. With a passive system a processor finishes one 
task before pesheerine the next, thus guaranteeing that task 
switching occurs at a time when there is little context, e.g., no 
resources are locked. 

Passive Systems are more. reliable Tor anouher reason: namely... 
the recovery mechanisms tend to be far simpler than those for ac- 
tive systems. 

As a third example of simplicity we introduce the notion of 
a reliability Subsystem. A reliability subsystem is a part of the 
overall system which is verified as a unit. A subsystem may in- 
clude a related set of hardware, program, and/or data structures. 
The boundaries of these reliability Subsystems are not necessarily 
related at all to the boundaries of the hardware subsystems (pro- 
cessors, busses, memories, etc.) described earlier. The entire 
system is broken into these subsystems, which verify one another 
in an orderly fashion. 

The subsystems are cleanly bounded with well-defined inter- 
faces. They are self-contained in that each includes a self-test 
mechanism and reset capability. They are isolated in that all com- 
munication between subsystems takes place passively via data struc- 
tures. Complete interlocking is provided at the boundary of every 
Subsystem so that the subsystems can operate asynchronously with 


respect to one another. 


14 


Ornstein 


The monitoring of one subsystem by another is performed using 
timer modules, as discussed below. These timer modules guarantee 
that the self-test mechanism of each subsystem operates, and this 
in turn guarantees that the entire subsystem is operating properly. 
Redundancy 

Redundancy is simultaneously a blessing and a curse. It 
occurs in the hardware and the software, and in both control and 
data paths. We deliverately introduce redundancy to provide relia- 
bility and to promote efficiency, and it frequently occurs because 
it is a natural way to build things. On the other hand the mere 
existence of redundancy implies a possible disagreement between the 
versions of the information. Such inconsistencies usually lead to 
erroneous, Dehnav Lor; and Of ven persist for Jone periods. 

It was not until we adopted a strategy of systematically 
searching out and identifying all the redundancy in every subsystem 
that we succeeded in making the subsystems reliable. This process 
thererlore Constitutes one of our einee basic strategies for con- 
structing robust software. 

We use the term redundancy here in a somewhat subtle sense, 
not only for cases in which the same information is stored in two 
places,. Dut also when: two ‘stored pleces: of information each amply 
a common fact although neither is necessarily sufficient to imply 
The Ocher. 


There are several methods of dealing with redundancy. The 
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first and best is to eliminate it, and always refer to a single 
copy of the information. When we choose not to eliminate it, we 
can check the redundancy and explicitly detect and correct any 
inconsistencies. It does not really matter how this is done as the 
system is recovering from a failure anyway. What is important is 
to resolve the inconsistency and keep the algorithm moving. Some- 
times. it 1s too difficult to test for inconsistency; then timers 
can be used as discussed in the next section. 
Let us consider a few examples of redundancy to make these 
1deas More -cOncrecve,. 
- A buffer holding a message to be processed, and a 

pointer to the buffer on a "to be processed" queue -- 

if the buffer and queue are inconsistent, the buffer will 

not be processed. Each buffer has its own timer and if 

not processed in a reasonable time, it will be replaced 

on the queue. 

A device requesting a bus cycle, and a request capturing 

flip-flop in the bus arbiter -- if the arbiter and device 

disagree, the bus may hang. A timer resets the bus after 

eae second of inactivity. | 

One processor seeing a memory word at a particular system 

address and another seeing the same word at the same ad- 

dress -- The software watches for inconsistencies and when 


a er 


they occur declares the memory or one of the processors 


16 


Ornstein 


unusable. 

The PID level used by a particular device and the device 

serviced in response to that level -- The PID level(s) 

used by each device are program-readable. A process periodi- 

cally reads them and forces the tables driving the program's 

response To agree. 
Timers 

We have adopted a uniform structure for implementing a 

monitoring function between reliability subsystems based on watch- 
dog timers. Consider a subsystem which is being monitored. We 
qdéestenm Such a SuUbSYSTEeEm To Cycle With a characteristLeo time ‘constant 
and insist that a complete self-consistency check be included with- 
an every cycle... Resular passace Through this cycle therefore is 
SULIT ACTeENG Tlncdicaviorm Or orrecy operation OF The Ssubsys vem. 27 
excessive time goes by without passage through the cycle, it implies 
that the subsystem has had a failure from which it has not been 
able to recover by itself. The mechanism for monitoring the cycle 
is a timer which is restarted by every passage through the cycle. 
We have both hardware and software timers ranging from five micro- 
seconds to two minutes in duration. Another subsystem can monitor 
ChAkSs: timer and take corrective. 4acti0n 11 20 ever runs out. To 
avoid the necessity for subsystems to be aware of one another's 
internal structure, each subsystem includes a reset mechanism which | 


may be externally activated. Thus corrective action consists 
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merely of invoking this reset. The reset algorithm is assumed to 
work although a particular incarnation in code may fail because 


it gets damaged. In such a case another subsystem (the code check- 


summer) will shortly repair the damage. 


Note that we have introduced an active element into our 
otherwise totally passive system. These resets constitute the 
only active elements and furthermore are invoked only after a 
failure has occurred. This approach seems to provide for the 
maximum isolation between subsystems. 

SYSTEM RELIABILITY STRUCTURE 

Lie eee previous section we described a mechanism whereby 
one subsystem can monitor another. Our system consists of a 
eer of subsystems in which each subsystem monitors the next 


member of the chain. Figure 1 and Table I show this structure in 


the system we have built for the IMP. An efficient way to build 


such. a chain is to have lower subsystems provide and guarantee 
some important environmental feature used by higher level systems. 
For example, a low level in our chain guarantees the integrity 

of code for higher levels which thus assume the correctness of 
code. Such a system is vulnerable only at its bottom. (We are 
assuming here that we have runnable hardware although it may be 
in a bad state, requiring resetting.) The code tester level 

(see Figure 1) serves three Panettenss first, it checksums all 


low level code (including itself); second, it insures that control 
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is operating properly, 1.e., that all subsystems are receiving a 
share of the processors! attention; third, it guarantees that locks 
do not hang up. It thus guarantees the most basic features for 

all higher levels. These will, in turn, provide further environ- 
mental features, such as a list of working memory areas, I/0 
devices, etec., to still higher levels. The method by which the 
code tester subsystem itself is aenivored and reset will be dis- 


cussed shortly. 


Table [I 
Major Subsystems and their Functions 
IMP SYSTEM: Watches network behavior - will not cooperate 
with irresponsible network behavior. 
IMP SYSTEM RELIABILITY: Watches IMP SYSTEM (data structures 
mostly). | 
CONSENSUS: Watches IMP SYSTEM RELIABILITY, verifies all Common 
Memory Code (via checksum), watches each processor, 
finds all usable hardware resources (interfaces, 
‘PIDs, memory , processors, etc.), tests each and 
creates a table of good ones. Makes spare copies 
of code. 
INDIVIDUAL: Watches CONSENSUS, finds 10) eenory and processors 


it considers usable, determines where the Consensus 
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CODE TESTER: Watches INDIVIDUAL, verifies all Local Memory Code 
(via a checksum), guarantees control and lock 
mechanisms« 

BUS TIMER + 60Hz INTERRUPT: Watches CODE TESTER, guarantees bus 


activity. 


The mechanisms we have described ensure that the separate 
processor subsystems have a -satistactory local environment in 
which to work. Before they can ore eopees LO. run the. main. 6ys— 
tem it is necessary that a common environment be established for 
all processors. We call the process of reaching an agreement 
about this environment "forming a consensus", and we dub the group 
of agreeing processors the Consensus. The work done by the Con- 
sensus is in fact performed by individual processors communicating 
Via common memory, but the coordination and discipline imposed on 
Consensus members make them behave like a single logical entity. 
An example of a task requiring consensus is the identification Or 
usable common memory and the assignment of functions (code, vari- 
ables, buffers, etc.) to particular pages. The members of the Con- 
sensus will not in general agree in piety view of the environment, 
as for example when a broken bus coupler blinds one member to a 
segment of common memory. In this case the Consensus, including 
one processor with the broken coupler, will agree to run the main 


SVSUCM. WLTHOUL That processor: 
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The Consensus maintains a igen: eon eis preeeneee at ene: 
system, whether or not the processor is won ie ‘The Consensus 
will count down these timers in order to steace uncooperative 
or dead pubeeesons. In order to join the Consensus, a processor 
need merely register its desire to join by holding off its timer. 
Within the individual processors it is the code tester subsystem 
which holds off the timer. 

The Consensus, then, acting as a group, provides the monitor- 
ing mechanism for the individuals as shown by the feedback monitor- 
ing path in Figure 1. This monitoring mechanism run by the Con- 
sensus includes the usual reset capability which in this case 
means reloading the individual's local memory and restarting the 
processor. Since all of the processors have identical memories, 
reloading is not difficult. We provide (password protected) paths 
whereby any processor can reset, reload, and restart any other 
processor. This reliance on the Consensus is indeed vulnerable to 
a Simultaneous transient failure of all processors. However, the 
Network Control Center has access to these same reset and reload 
facilities and these enable it to perform the reload function re- 
motely (a path also shown in the figure). 

Thus the eonsensus and/or Network Control Center are the 
ultimate guarantors of the lowest level subsystem. While this 
process is sufficient it is sometimes slow. For many cases in 


which the Consensus is disabled (as for example when all of the 
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processors halt), a simpler reset without reloading will suffice. 
For this reason we have provided a simpler and more immediate (if 
redundant) mechanism in each processor for resetting the control 
and lock systems. We implement this mechanism in software with 

the assistance of a 60Hz interrupt and a one-second timer on the 
bus. Together these provide a somewhat vulnerable but much quicker 
alternative to the more ponderous NCC/Consensus resets. 

There is a problem about what area of common memory the 
processors should use in which to form the Consensus, since 
failures may make any predetermined system address inaccessible. 
To allow for this, sufficient communication is maintained in all 
pages of common memory to reach agreement both as to which proces- 
Sors. are in the Consensus and where further communication: 1s to 
take place. 

TO Provect. 127sellt trom broken processors, “bhe Consensus. ampu= 
tates all processors which do not succeed in joining it. There is 
a conflict between this need to protect itself and the need to 
admit new or healed processors into the Consensus. The snpuEAe iON 
barrier is therfore lowered for a brief period each time the Con- 
SENSUS TY1e6S, tO restart. a processor. This Vestvart: 48 in. fact the 
reset... based On: the timer: Neild off by the code ‘Lester subsystem. 
as discussed above. In the case of certain active failures, even 
this brief relaxation may cause trouble. In these cases the Con- 


sensus will decide to keep the barrier up continuously. 
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Certain active failures may prevent the formation of a con- 
cee In such a situation each processor will behave as if it 
were a Consensus (of one) and will try to amputate all other proces~ 
sors. At the point when the actively failing component is ampu- 
tated, the remaining processors will be able to form a consensus. 
From this point the system behaves as described above. 

Further up in the figure there is another joining of inde- 
pendent units, namely IMPs joining to form the network. The 
analogy here is incomplete because the ARPA Network was not built 
with these concepts in mind. There Me COLLECCAVe:. DERAVIOr. -Ewe sg 
routing; and: individual. behavior which: -accepus. collective 
decisions only after they pass reasonability tests. However, the 
reliability features of the network are concentrated in the Net- 
work Control Center, which depends on the continual presence of 
human operators for successful operation. It is correspondingly 
powerful, resourceful, and erratic in its behavior. 

SOME EXAMPLES OF FATLURES 

In order to explain in more practical terms some of the 
reliability mechanisms, we now discuss a number of specific failures 
and describe the methods which detect and repair the resulting 
damage. In each case, we focus on the component that failed and 
the particular mechanism that takes care of that failure. Deriva- 
tive failures may well take place, and other mechanisms will handle 


these, since all mechanisms operate all the time. 
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These examples are set in the context of the IMP application and 
the severity of their direct consequences rated on the following 
scale: 

1. Momentary slowdown - no data loss 

2. Loss of data (a network message) 

3. Temporary loss of some IMP function (a network link) 

4, Momentary total IMP outage with local self-recovery 

5. Outage requiring reloading via the network 

6 Failure requiring human insight for debugging. 
Fxample 1. Suppose first that a bus coupler experiences a transient 
failure on a single reference to common memory, which leaves one 
noee of common memory with the wrong contents but correct parity. 
Suppose further that the failure is subtle, in the sense that there 
is no obvious ill effect on processor control, like halting or 
looping, which will be caught by lower level mechanisms. We will 
focus first on examples which cause minimal disruption and where 
detection and gentle recovery are the primary concerns. We con- 
sider four examples of transient memory failures: 
Example l.a Suppose that a word of text in one of the messages 
we are deliverying becomes smashed. There is a checksum on all 
messages and the network will notice at one of its checkpoints that 
the nessage has gone bad. The source will be prompted to send a 


new copy. (Severity 2) 
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Example 1.b Near the heart of our system is a queue of unused 
buffers called the free list. Suppose the failure is in the struc- 
ture of this queue. The system explicitly tests for both a ieeces 
queue and wrong things on the queue. A more subtle form of error 
occurs when some buffers which should be on the queue are missing 
from it. Our system is designed so that a buffer should be removed 
from the free list for at most two minutes at a time. A timer is 
maintained on each buffer, which is restarted whenever the buffer 
returns to the free list. Should any timer ever run out, its 
buffer is forced back onto the free list. The result Of thLS 
failure will be a degradation of system performance as it attempts 
to run with fewer buffers for a short while, followed by complete 
recovery within two minutes. The IMP will stay up and no messages 
will be lost. (Severity 1) 

Example l.c Suppose that one of the locks on a resource is 

broken so that it wrongly locks the resource. Any subsystem which 
tries to use ‘the resource will put a processor into a tight loop 
waiting for the resource to become free. In about 1/15 sec. this 
will cause the processor's timer, driven off its 60Hz clock inter- 
PUD. to run out. Upon investigation, the program will notice that 
the subsystem is waiting for a locked resource. and: arpitraraly 
Unlock. its. Aside. from the 1715 sec. pause, ‘the system willbe 
unaffected by the transient. (Compare the simplicity of this scheme 


with 14) (Severity 1) 
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Example 1.d Suppose now that a failure strikes common memory 
holding code, and that the trouble is subtle -- either the code 

is not run often or the change has no immediate drastic effect. 

In a few seconds the processors will begin to notice that the 
cheeksum on Thay. phece: or code 128 bad. and; Stcp running 16: Shortly 
the whole Consensus will agree, and will switch over to use the 
memory holding the spare copy of that code. Unless the broken 

code has already caused some other trouble, the problem is thereby 
fixed, with only momentary slowdown. (Severity 1) 

Example 2. Suppose a processor fails by suddenly and permanently 
stopping. The immediate effect will be that some task will be left 
half done, with a high probability that some resource is locked. 
TOUS 1O0ks TO the system Like 4 data: Tatlure,. 28 2n:-exanptles. 1.2. 
Lad, and -ac- above. The recovery wild. be tdentical.. Ina few 
seconds the Consensus will notice that the DPrOCeESSOr: nas dropped 
out and processor recovery logic will be invoked. Since the 
processor is SOlLIdLY broken tne recovery will pe unsuccessiuls and 
the system will settle into a mode where every so often recovery 

is retried. Eventually a repairman will fix the processor, at 
which time recovery will proceed and the processor will rejoin Con- 
sensus. It is hard to predict whether the IMP system will momen- 
tarily go down because of the failure; experience indicates that 

EC USUaGLIY ‘Stays. Up, DU. Our Experience Ls. Timited to. lieholy 


loaded machines. (Severity 2-4) 
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Example 3. Sunovose a power supply for a processor bus breaks. | 
This is similar to the failing processor described above except 
that both processors on the bus are affected and the processors 
are given a hardware warning sufficiently far in advance that they 
can halt cleanly. The system will surely stay up through this 
failures Leeverity 1) 
Example 4. Now consider a case in which some page of common memory 
ceases to answer when referenced. Each processor will get a hard- 
ware trap when it tries to use that memory, forcing it directly to 
the code which routinely verifies the environment. As a result, the 
failing memory will be deleted from the memory Mise by the Consensus 
and another module will be pressed into service to take its place. 
if the tailed page contained. Code, a Spare copy will normally 
be available and a new spare copy will be made if possible. a 
it contained data, an unused page will be pressed into service. 
In either case, the system will be reinitialized, momentarily 
bringing the IMP system down. If the failed page contained the 
Consensus communication area, a new Consensus must be formed and 
thus recovery will take a little longer. (Severity 4) 
Example 5. Let uS now consider a failure of the PID. Suppose that 
the PID reports a task not previously set. When invoked, each strip 
checks to make sure that it is reasonable for the strip to be run. 
If not, another task is sought. Suppose instead that the PID "drops" 
a task. A special process Semicatoalty sets all PID flags inde- 


pendent of what needs to be done. This causes no harm, because 
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superfluous tasks will be ignored (as described above), and serves 
to pick up such dropped tasks. Thus we have both a consistency 
eheck on redundant information and a timer built into our use of 
the PID. If a PID fails solidly, another PID will be switched in 
to operate the system. Transient failures cause slowdown; switch- 
over may momentarily bring down the IMP system. (Severity 1, 4) 

All ol thas leads to-a silaently-dirffrerent. image of the PID: 
Instead of being the central task disburser, with all processors 
relyine On 20 CO vel Chem whew to -d0e, the chp: as a pulde. .suge 
PesSting to processors that if they look in 4 certein place, they 
will probably find some useful work to do. The system would in 
fact: Pun. wienouy:a@- PID. albert much more Siowky -and. inet riciencly.« 
Example 6. Suppose a halt instruction somehow gets planted in 
COMMON. Memory wand that. all processors: ‘execute 10 and. Slop. “There 
LS. Tous no Consensus: Jert- U0 «come to the rescue, BPurthernore, 
60Hz interrupts are ineffective in a halted processor. After one 
second of inactivity, the bus arbiter timer will reset the proces- 
sors, making them once more eligible for 60Hz interrupts which will 
restart them. Before the broken code 1s run, it will be checksummed, 
the discrepancy found, and a spare copy used. (Severity 2-4) 
Example 7. . Let us consider now what happens when, in common memory, 
an end test for a storing loop is destroyed, causing each processor _ 


to wipe out its 60Hz interrupt code in local memory. In this case 
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not only are there no processors left to help, but the 60Hz inter- 
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broken. This is a case in which the machine is incapable of res- 
cuing itself and will go off the ee as a working node. When 
the Network Control Center notices that the IMP is no longer up, 
it will commence an external reload, restoring the IMP to operation. 
(Severity 5) 

Example 8. Consider the case of a processor whose hardware is 
solidly broken such that it eeopatcais Spores a zero into a loca- 
GrOM 2n common memory. Mechanisms described above will repeatedly 
fix the changed location, but it is desirable to eliminate the 
continuing presence of this disrupting influence. The Consensus 
will notice that one of its number has dropped out and will endeavor 
to help the event processor. After a few tries, a longer timer 
will run out, and the Consensus will take a more: Gdrastic.actvi0n= 
final amputation. In this case Gene will be a rather lengthy 

IMP outage but the system will recover without external help. 
(Severity 4) 

Example 9. One failure from which there is no recovery, either 
automatic or remote, is a program which impersonates normal behavior 
‘but is still somehow incorrect. That is,-it holds off the right 
timers, has a valid checksum, and simulates enough normal behavior 
so that higher levels (e.g., the NCC) are satisfied. For example, 
if it were not for the fact that the NCC explicitly checks the 
version vumber of the program running in each IMP, a previous 
compatible, but obsolete version of the program would exhibit this 


behavior. (Severity 6) 
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Example 10. Another Class of Tailures which 1s hard to isolate 
and deal with is low-frequency intermittents. Consider the case 
of a single processor which is broken such that its indexed Shit t 
INSeErVUCL LON: Periorms: 2necorrectly«. sinee This instrucrion. only 
occurs in some infrequently executed procedures, the failure only 


ManiTests. LUSeld ~ On tne average. once every period tx 11 “Gb as 


large, for instance one year, then we can safely disregard the error, 


since its frequency is in the range of other failures over which 
we have no control. If t 18 small, say LOO milliseconds, then the 
Consensus will isolate the bad processor and excise it. At some 
intermediate frequency, however, the Consensus will fail tc cor- 
relate successive failures and will instead treat each as a separate 
bransient. Whe system wilh repearvediy tail and Pecover until some 
human intervenes. (Severity 6) 

RESULTS AND CONCLUSIONS 

Some strategies and techniques for building a reliable multi- 
processor have been described above. We have, in fact, actually 
built and programmed such a machine uSing these strategies. We 
have found this machine straightforward to debug, both in hardware 
and software. Furthermore, the system continues to operate when 
CLR ea power supplies are turned off, when memory locations 
are altered, when cables and cards are torn out, and through a 
variety of other failures. We have yet to establish field per- 


formance (which must be measured both in rate of recoverable 
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incidents and in rate of unrecoverable failures), but we expect to 
start gathering this information Shortly. 

We believe there are many important problems in the world 
today which could benefit from the principles described here. 
While we have discussed these principles in terms of a specific 
application (the IMP), most of the concepts are application inde- 
pendent. We have been able to separate the application code from 
the reliability subsystems intact in another application of the 
Pluribus machine. 
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